Sep 15 18:48:52.455625 ip-10-0-132-28 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 15 18:48:52.455640 ip-10-0-132-28 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 15 18:48:52.455649 ip-10-0-132-28 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 15 18:48:52.455903 ip-10-0-132-28 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 15 18:49:02.511659 ip-10-0-132-28 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 15 18:49:02.511682 ip-10-0-132-28 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 4a04ce91424f47cbaa0c11639f3294f9 --
Sep 15 18:50:58.393899 ip-10-0-132-28 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 18:50:58.819556 ip-10-0-132-28 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 18:50:58.819556 ip-10-0-132-28 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 18:50:58.819556 ip-10-0-132-28 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 18:50:58.819556 ip-10-0-132-28 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 18:50:58.819556 ip-10-0-132-28 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 18:50:58.820640 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.820537    2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 18:50:58.825127 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825098    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:58.825127 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825126    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:58.825127 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825131    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825134    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825137    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825141    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825143    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825147    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825150    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825152    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825155    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825158    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825162    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825164    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825167    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825170    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825173    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825175    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825198    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825201    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825204    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825207    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:58.825230 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825210    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825213    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825216    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825218    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825221    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825224    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825226    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825229    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825232    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825235    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825239    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825242    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825245    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825247    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825250    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825253    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825257    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825260    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825262    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825265    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:58.825733 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825268    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825270    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825273    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825275    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825278    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825280    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825283    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825285    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825288    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825290    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825293    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825295    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825298    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825301    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825304    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825307    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825310    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825312    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825315    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825317    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:58.826219 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825320    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:58.826722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825323    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:58.826722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825325    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:58.826722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825328    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:58.826722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825330    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:58.826722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825333    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:58.826722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825335    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:58.826722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825338    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:58.826722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825341    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:58.826722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825344    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:58.826722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825346    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:58.826722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825349    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:58.826722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825351    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:58.826722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825354    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:58.826722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825357    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:58.826722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825359    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:58.826722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825362    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:58.826722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825366    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:58.826722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825370    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:58.826722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825372    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:58.827217 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825375    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:58.827217 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825378    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:58.827217 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825380    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:58.827217 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825384    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:58.827217 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825387    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:58.827217 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825389    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:58.827217 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825394    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:58.827217 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825397    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:58.827217 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825399    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:58.827217 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825402    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:58.827217 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825405    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:58.827217 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825409    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:58.827217 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825412    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:58.827217 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825416    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:58.827217 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825418    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:58.827217 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825421    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:58.827217 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.825425    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:58.827217 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826192    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:58.827217 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826201    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:58.827775 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826221    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:58.827775 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826225    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:58.827775 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826229    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:58.827775 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826233    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:58.827775 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826236    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:58.827775 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826239    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:58.827775 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826242    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:58.827775 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826245    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:58.827775 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826248    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:58.827775 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826250    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:58.827775 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826253    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:58.827775 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826256    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:58.827775 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826258    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:58.827775 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826261    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:58.827775 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826264    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:58.827775 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826267    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:58.827775 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826271    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:58.827775 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826275    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:58.827775 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826278    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826281    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826285    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826288    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826291    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826294    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826296    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826299    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826301    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826304    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826307    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826309    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826312    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826314    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826317    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826319    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826321    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826324    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826327    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826330    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:58.828255 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826332    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:58.828763 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826335    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:58.828763 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826337    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:58.828763 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826340    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:58.828763 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826342    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:58.828763 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826345    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:58.828763 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826347    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:58.828763 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826350    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:58.828763 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826352    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:58.828763 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826357    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:58.828763 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826360    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:58.828763 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826363    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:58.828763 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826366    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:58.828763 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826369    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:58.828763 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826372    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:58.828763 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826375    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:58.828763 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826377    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:58.828763 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826380    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:58.828763 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826382    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:58.828763 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826385    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826387    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826391    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826393    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826396    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826399    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826402    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826405    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826407    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826410    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826412    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826415    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826418    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826421    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826423    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826426    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826428    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826432    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826434    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826437    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:58.829244 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826440    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826442    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826446    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826448    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826451    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826453    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826456    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826459    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826461    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826464    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826467    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826469    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826473    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826476    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826479    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826481    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826484    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826487    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826489    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826491    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:58.829753 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826885    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826890    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826893    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826896    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826899    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826902    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826904    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826907    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826910    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826913    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826916    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826919    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826921    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826924    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826927    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826930    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826932    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826935    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826938    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826940    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:58.830251 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826943    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826945    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826948    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826951    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826953    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826956    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826958    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826961    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826963    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826966    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826968    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826971    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826973    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826976    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826978    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826982    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826985    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826988    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826991    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826993    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:58.830770 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826995    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:58.831258 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.826998    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:58.831258 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827002    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:58.831258 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827004    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:58.831258 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827007    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:58.831258 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827010    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:58.831258 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827012    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:58.831258 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827015    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:58.831258 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827018    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:58.831258 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827020    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:58.831258 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827023    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:58.831258 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827026    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:58.831258 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827028    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:58.831258 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827031    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:58.831258 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827033    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:58.831258 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827036    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:58.831258 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827040    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:58.831258 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827044    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:58.831258 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827047    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:58.831258 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827049    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827053    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827055    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827058    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827061    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827064    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827067    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827069    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827071    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827074    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827076    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827079    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827082    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827084    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827087    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827090    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827092    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827095    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827097    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827100    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:58.831737 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827103    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827105    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827108    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827110    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827113    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827115    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827118    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827121    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827123    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827126    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827128    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827132    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827134    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827137    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827141    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827144    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827147    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827149    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827152    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827155    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:58.832269 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827229    2555 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827266    2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827272    2555 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827277    2555 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827282    2555 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827286    2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827290    2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827295    2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827299    2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827302    2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827305    2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827313    2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827316    2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827319    2555 flags.go:64] FLAG: --cgroup-root=""
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827322    2555 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827326    2555 flags.go:64] FLAG: --client-ca-file=""
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827328    2555 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827332    2555 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827337    2555 flags.go:64] FLAG: --cluster-domain=""
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827340    2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827343    2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827346    2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827350    2555 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827354    2555 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 18:50:58.832769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827357    2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827360    2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827364    2555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827367    2555 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827371    2555 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827374    2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827377    2555 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827380    2555 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827384    2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827387    2555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827390    2555 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827393    2555 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827396    2555 flags.go:64] FLAG: --enable-server="true"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827400    2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827405    2555 flags.go:64] FLAG: --event-burst="100"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827408    2555 flags.go:64] FLAG: --event-qps="50"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827411    2555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827414    2555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827417    2555 flags.go:64] FLAG: --eviction-hard=""
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827427    2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827430    2555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827433    2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827436    2555 flags.go:64] FLAG: --eviction-soft=""
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827439    2555 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827442    2555 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 18:50:58.833361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827446    2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827448    2555 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827452    2555 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827454    2555 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827458    2555 flags.go:64] FLAG: --feature-gates=""
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827462    2555 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827465    2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827470    2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827473    2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827476    2555 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827480    2555 flags.go:64] FLAG: --help="false"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827483    2555 flags.go:64] FLAG: --hostname-override="ip-10-0-132-28.ec2.internal"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827486    2555 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827489    2555 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827491    2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827495    2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827498    2555 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827501    2555 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827504    2555 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827508    2555 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827511    2555 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827515    2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827518    2555 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827521    2555 flags.go:64] FLAG: --kube-reserved=""
Sep 15 18:50:58.834055 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827524    2555 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827527    2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827531    2555 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827534    2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827537    2555 flags.go:64] FLAG: --lock-file=""
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827540    2555 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827543    2555 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827546    2555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827552    2555 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827555    2555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827558    2555 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827561    2555 flags.go:64] FLAG: --logging-format="text"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827564    2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827567    2555 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827570    2555 flags.go:64] FLAG: --manifest-url=""
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827575    2555 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827580    2555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827583    2555 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827587    2555 flags.go:64] FLAG: --max-pods="110"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827591    2555 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827594    2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827597    2555 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827600    2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827603    2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827620    2555 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 18:50:58.834704 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827624    2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827631    2555 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827634    2555 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827637    2555 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827640    2555 flags.go:64] FLAG: --pod-cidr=""
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827643    2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827649    2555 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827652    2555 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827655    2555 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827658    2555 flags.go:64] FLAG: --port="10250"
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827662    2555 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827665    2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a062855a91710799"
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827668    2555 flags.go:64] FLAG: --qos-reserved=""
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827671    2555 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827674    2555 flags.go:64] FLAG: --register-node="true"
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827677    2555 flags.go:64] FLAG: --register-with-taints=""
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827681    2555 flags.go:64] FLAG: --registry-burst="10"
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827684    2555 flags.go:64] FLAG: --registry-qps="5"
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827687    2555 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827690    2555 flags.go:64] FLAG: --reserved-memory=""
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827694    2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827697    2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827702    2555 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827705    2555 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827708    2555 flags.go:64] FLAG: --runonce="false"
Sep 15 18:50:58.835332 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827711    2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827714    2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827717    2555 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827720    2555 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827723    2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827726    2555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827729    2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827732    2555 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827735    2555 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827738    2555 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827741    2555 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827743    2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827746    2555 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827750    2555 flags.go:64] FLAG: --system-cgroups=""
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827753    2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827758    2555 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827761    2555 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827765    2555 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827771    2555 flags.go:64] FLAG: --tls-min-version=""
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827774    2555 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827777    2555 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827780    2555 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827783    2555 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827786    2555 flags.go:64] FLAG: --v="2"
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827790    2555 flags.go:64] FLAG: --version="false"
Sep 15 18:50:58.836000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827794    2555 flags.go:64] FLAG: --vmodule=""
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827798    2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.827802    2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827932    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827938    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827941    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827944    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827947    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827950    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827953    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827956    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827960    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827962    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827965    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827968    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827972    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827975    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827977    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827980    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827983    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:58.836598 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827985    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:58.837121 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827989    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:58.837121 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827991    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:58.837121 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827994    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:58.837121 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.827997    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:58.837121 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828000    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:58.837121 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828002    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:58.837121 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828005    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:58.837121 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828007    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:58.837121 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828010    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:58.837121 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828013    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:58.837121 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828015    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:58.837121 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828018    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:58.837121 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828020    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:58.837121 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828023    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:58.837121 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828025    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:58.837121 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828030    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:58.837121 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828032    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:58.837121 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828035    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:58.837121 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828037    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828040    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828043    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828045    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828047    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828050    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828052    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828055    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828057    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828060    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828062    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828065    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828067    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828070    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828073    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828083    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828086    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828089    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828091    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828094    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:58.837597 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828096    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828099    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828101    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828104    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828107    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828110    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828112    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828115    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828117    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828121    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828124    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828127    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828129    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828132    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828134    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828137    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828140    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828142    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828145    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828148    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:58.838113 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828150    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:58.838692 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828153    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:58.838692 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828156    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:58.838692 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828159    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:58.838692 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828161    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:58.838692 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828164    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:58.838692 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828166    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:58.838692 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828169    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:58.838692 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828172    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:58.838692 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828175    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:58.838692 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828177    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:58.838692 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828180    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:58.838692 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828182    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:58.838692 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828185    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:58.838692 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828187    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:58.838692 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828190    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:58.838692 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828194    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:58.838692 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828198    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:58.838692 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828202    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:58.838692 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828204    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:58.839204 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828207    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:58.839204 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828212    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:58.839204 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.828215    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:58.839204 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.828853    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 18:50:58.839204 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.835907    2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 18:50:58.839204 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.835923    2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 18:50:58.839204 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.836983    2555 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 18:50:58.839204 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.836992    2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 18:50:58.839204 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837051    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:58.839204 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837058    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:58.839204 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837061    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:58.839204 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837064    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:58.839204 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837067    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:58.839204 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837070    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:58.839204 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837073    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837076    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837078    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837081    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837084    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837086    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837089    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837092    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837094    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837099    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837102    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837105    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837108    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837119    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837122    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837125    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837128    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837131    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837134    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837137    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:58.839579 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837140    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837143    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837145    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837150    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837153    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837156    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837159    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837162    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837164    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837167    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837170    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837172    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837175    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837178    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837180    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837183    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837186    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837188    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837191    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837193    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:58.840096 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837196    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:58.840596 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837199    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:58.840596 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837201    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:58.840596 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837204    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:58.840596 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837206    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:58.840596 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837209    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:58.840596 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837213    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:58.840596 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837217    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:58.840596 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837220    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:58.840596 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837222    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:58.840596 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837227    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:58.840596 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837230    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:58.840596 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837233    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:58.840596 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837236    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:58.840596 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837239    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:58.840596 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837242    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:58.840596 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837246    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:58.840596 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837249    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:58.840596 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837252    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:58.840596 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837254    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837257    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837260    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837263    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837265    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837268    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837270    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837273    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837276    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837279    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837281    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837284    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837286    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837289    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837291    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837294    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837297    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837299    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837302    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837304    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:58.841087 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837307    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:58.841582 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837310    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:58.841582 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837312    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:58.841582 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837314    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:58.841582 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837317    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:58.841582 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837320    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:58.841582 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837322    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:58.841582 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837325    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:58.841582 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837328    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:58.841582 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837331    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:58.841582 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837334    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:58.841582 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837337    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:58.841582 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837340    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:58.841582 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837343    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:58.841582 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.837348    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 18:50:58.841582 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837461    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837467    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837470    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837473    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837477    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837482    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837485    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837488    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837491    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837493    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837496    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837499    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837502    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837504    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837506    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837509    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837512    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837514    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837517    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837520    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:58.841976 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837522    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837525    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837527    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837530    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837532    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837535    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837538    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837541    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837544    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837546    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837549    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837551    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837554    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837556    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837559    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837561    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837564    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837566    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837569    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837572    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:58.842476 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837574    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837576    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837579    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837582    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837584    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837587    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837589    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837592    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837594    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837597    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837600    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837602    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837620    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837624    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837628    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837631    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837634    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837636    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837639    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837642    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:58.843020 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837644    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:58.843534 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837648    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:58.843534 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837651    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:58.843534 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837654    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:58.843534 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837656    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:58.843534 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837659    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:58.843534 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837662    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:58.843534 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837665    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:58.843534 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837669    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:58.843534 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837672    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:58.843534 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837675    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:58.843534 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837677    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:58.843534 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837680    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:58.843534 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837683    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:58.843534 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837685    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:58.843534 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837688    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:58.843534 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837691    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:58.843534 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837693    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:58.843534 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837696    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:58.843534 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837698    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:58.844015 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837702    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:58.844015 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837705    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:58.844015 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837708    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:58.844015 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837711    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:58.844015 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837713    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:58.844015 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837716    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:58.844015 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837719    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:58.844015 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837721    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:58.844015 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837724    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:58.844015 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837727    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:58.844015 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837729    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:58.844015 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837732    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:58.844015 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837735    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:58.844015 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837737    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:58.844015 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837740    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:58.844015 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837743    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:58.844015 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837745    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:58.844015 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837748    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:58.844015 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:50:58.837750    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:58.844474 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.837755    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 18:50:58.844474 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.837878    2555 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 18:50:58.844474 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.839872    2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 18:50:58.844474 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.840906    2555 server.go:1019] "Starting client certificate rotation"
Sep 15 18:50:58.844474 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.841009    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 18:50:58.844474 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.841043    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 18:50:58.844474 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.843479    2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 18:50:58.848667 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.848639    2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 18:50:58.860970 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.860944    2555 log.go:25] "Validated CRI v1 runtime API"
Sep 15 18:50:58.864746 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.864730    2555 log.go:25] "Validated CRI v1 image API"
Sep 15 18:50:58.865590 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.865572    2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 18:50:58.868179 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.868153    2555 fs.go:135] Filesystem UUIDs: map[1fc874e3-e0cc-4b50-b9e4-2010ea7bfc67:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fa528439-f6f7-435a-b537-cc8624dfe3a0:/dev/nvme0n1p3]
Sep 15 18:50:58.868264 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.868175    2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 18:50:58.872807 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.872790    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 18:50:58.873350 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.873238    2555 manager.go:216] Machine: {Timestamp:2026-09-15 18:50:58.871845943 +0000 UTC m=+0.359174918 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097100 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28adfd8e807ed23f4f86abd8c01853 SystemUUID:ec28adfd-8e80-7ed2-3f4f-86abd8c01853 BootID:4a04ce91-424f-47cb-aa0c-11639f3294f9 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c8:61:14:4d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c8:61:14:4d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:28:85:c9:20:bc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 18:50:58.873350 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.873349    2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 18:50:58.873449 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.873433    2555 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 18:50:58.876602 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.876565    2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 18:50:58.876791 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.876604    2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-28.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 18:50:58.877365 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.877353    2555 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 18:50:58.877424 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.877370    2555 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 18:50:58.877424 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.877381    2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 18:50:58.877971 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.877959    2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 18:50:58.878724 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.878713    2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 18:50:58.883955 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.883941    2555 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 18:50:58.884086 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.884075    2555 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 18:50:58.884142 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.884117    2555 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 18:50:58.884142 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.884131    2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 18:50:58.884234 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.884150    2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 18:50:58.884234 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.884164    2555 kubelet.go:398] "Adding apiserver pod source"
Sep 15 18:50:58.884234 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.884176    2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 18:50:58.884565 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.884551    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 18:50:58.884660 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.884574    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 18:50:58.886924 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.886908    2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 18:50:58.889472 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.889457    2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 18:50:58.889552 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.889487    2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 18:50:58.890877 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.890863    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 18:50:58.890877 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.890880    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 18:50:58.890983 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.890886    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 18:50:58.890983 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.890892    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 18:50:58.890983 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.890900    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 18:50:58.890983 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.890906    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 18:50:58.890983 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.890919    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 18:50:58.890983 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.890924    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 18:50:58.890983 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.890930    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 18:50:58.890983 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.890937    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 18:50:58.890983 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.890945    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 18:50:58.890983 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.890950    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 18:50:58.891654 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.891643    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 18:50:58.891703 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.891655    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 18:50:58.892746 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:58.892725    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 18:50:58.892834 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:58.892816    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 18:50:58.895541 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.895526    2555 server.go:1268] "Started kubelet"
Sep 15 18:50:58.895665 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.895627    2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 18:50:58.895719 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.895654    2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 18:50:58.895765 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.895742    2555 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 18:50:58.899800 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.899776    2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 18:50:58.900083 ip-10-0-132-28 systemd[1]: Started Kubernetes Kubelet.
Sep 15 18:50:58.901167 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.900560    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j2xpl"
Sep 15 18:50:58.902009 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.901696    2555 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 18:50:58.903049 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:58.903027    2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 18:50:58.905161 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.905142    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j2xpl"
Sep 15 18:50:58.906498 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:58.905496    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-28.ec2.internal.18d5932f6fa55bce  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-28.ec2.internal,UID:ip-10-0-132-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-28.ec2.internal,},FirstTimestamp:2026-09-15 18:50:58.895494094 +0000 UTC m=+0.382823069,LastTimestamp:2026-09-15 18:50:58.895494094 +0000 UTC m=+0.382823069,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-28.ec2.internal,}"
Sep 15 18:50:58.906903 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.906886    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 18:50:58.907443 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.907425    2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 18:50:58.907594 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:58.907571    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-28.ec2.internal\" not found"
Sep 15 18:50:58.908149 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.908081    2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 18:50:58.908149 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.908084    2555 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 18:50:58.908149 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.908124    2555 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 18:50:58.908320 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.908224    2555 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 18:50:58.908320 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.908234    2555 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 18:50:58.908429 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.908411    2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 18:50:58.908429 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.908428    2555 factory.go:55] Registering systemd factory
Sep 15 18:50:58.908526 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.908435    2555 factory.go:223] Registration of the systemd container factory successfully
Sep 15 18:50:58.908642 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.908627    2555 factory.go:153] Registering CRI-O factory
Sep 15 18:50:58.908700 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.908645    2555 factory.go:223] Registration of the crio container factory successfully
Sep 15 18:50:58.908700 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.908662    2555 factory.go:103] Registering Raw factory
Sep 15 18:50:58.908700 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.908671    2555 manager.go:1196] Started watching for new ooms in manager
Sep 15 18:50:58.909071 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.909058    2555 manager.go:349] Starting recovery of all containers
Sep 15 18:50:58.912540 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.912521    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:58.916105 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:58.916082    2555 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-28.ec2.internal\" not found" node="ip-10-0-132-28.ec2.internal"
Sep 15 18:50:58.917500 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.917476    2555 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-132-28.ec2.internal" not found
Sep 15 18:50:58.918829 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.918694    2555 manager.go:354] Recovery completed
Sep 15 18:50:58.923291 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.923276    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:58.925512 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.925491    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:58.925597 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.925529    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:58.925597 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.925544    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:58.926255 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.926238    2555 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 18:50:58.926255 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.926251    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:58.926380 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.926256    2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 18:50:58.926380 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.926276    2555 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 18:50:58.927023 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.927008    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:58.927098 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.927031    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:58.927098 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.927040    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:58.928401 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.928389    2555 policy_none.go:49] "None policy: Start"
Sep 15 18:50:58.928436 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.928405    2555 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 18:50:58.928436 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.928415    2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 18:50:58.930371 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.930358    2555 policy_none.go:47] "Start"
Sep 15 18:50:58.934988 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.934969    2555 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-132-28.ec2.internal" not found
Sep 15 18:50:58.956458 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.956445    2555 manager.go:337] "Starting Device Plugin manager"
Sep 15 18:50:58.956534 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:58.956483    2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 18:50:58.956534 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.956494    2555 server.go:87] "Starting device plugin registration server"
Sep 15 18:50:58.956890 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.956873    2555 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 18:50:58.956996 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.956891    2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 18:50:58.957281 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.957093    2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 18:50:58.957281 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.957188    2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 18:50:58.957281 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.957197    2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 18:50:58.957904 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:58.957885    2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 18:50:58.957999 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:58.957971    2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-28.ec2.internal\" not found"
Sep 15 18:50:58.991585 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:58.991558    2555 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-132-28.ec2.internal" not found
Sep 15 18:50:59.031972 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.031942    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 18:50:59.033208 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.033190    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 18:50:59.033271 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.033247    2555 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 18:50:59.033363 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.033271    2555 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 18:50:59.033408 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:59.033375    2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 18:50:59.035655 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.035635    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:59.057521 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.057500    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:59.058441 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.058426    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:59.058529 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.058453    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:59.058529 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.058465    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:59.058529 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.058493    2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-28.ec2.internal"
Sep 15 18:50:59.064156 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.064131    2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-28.ec2.internal"
Sep 15 18:50:59.064156 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:59.064163    2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-28.ec2.internal\": node \"ip-10-0-132-28.ec2.internal\" not found"
Sep 15 18:50:59.082808 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:59.082728    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-28.ec2.internal\" not found"
Sep 15 18:50:59.134294 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.134257    2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-28.ec2.internal"]
Sep 15 18:50:59.134453 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.134379    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:59.136053 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.136036    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:59.136157 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.136068    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:59.136157 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.136083    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:59.138389 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.138374    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:59.138510 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.138493    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal"
Sep 15 18:50:59.138571 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.138530    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:59.139310 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.139292    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:59.139418 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.139321    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:59.139418 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.139330    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:59.139418 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.139297    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:59.139418 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.139405    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:59.139418 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.139421    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:59.141516 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.141500    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-28.ec2.internal"
Sep 15 18:50:59.141590 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.141523    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:59.142235 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.142222    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:59.142324 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.142246    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:59.142324 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.142256    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:59.157142 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:59.157123    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-28.ec2.internal\" not found" node="ip-10-0-132-28.ec2.internal"
Sep 15 18:50:59.160463 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:59.160446    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-28.ec2.internal\" not found" node="ip-10-0-132-28.ec2.internal"
Sep 15 18:50:59.182842 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:59.182824    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-28.ec2.internal\" not found"
Sep 15 18:50:59.209645 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.209621    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44b7a7d82c3ab405a9df12dc028af9e9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal\" (UID: \"44b7a7d82c3ab405a9df12dc028af9e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal"
Sep 15 18:50:59.209725 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.209651    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44b7a7d82c3ab405a9df12dc028af9e9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal\" (UID: \"44b7a7d82c3ab405a9df12dc028af9e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal"
Sep 15 18:50:59.209725 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.209669    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f8d47b24220b0dba80a2c6cc3e11210a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-28.ec2.internal\" (UID: \"f8d47b24220b0dba80a2c6cc3e11210a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-28.ec2.internal"
Sep 15 18:50:59.283096 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:59.283049    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-28.ec2.internal\" not found"
Sep 15 18:50:59.310430 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.310404    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44b7a7d82c3ab405a9df12dc028af9e9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal\" (UID: \"44b7a7d82c3ab405a9df12dc028af9e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal"
Sep 15 18:50:59.310511 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.310432    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44b7a7d82c3ab405a9df12dc028af9e9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal\" (UID: \"44b7a7d82c3ab405a9df12dc028af9e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal"
Sep 15 18:50:59.310511 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.310451    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f8d47b24220b0dba80a2c6cc3e11210a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-28.ec2.internal\" (UID: \"f8d47b24220b0dba80a2c6cc3e11210a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-28.ec2.internal"
Sep 15 18:50:59.310583 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.310519    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44b7a7d82c3ab405a9df12dc028af9e9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal\" (UID: \"44b7a7d82c3ab405a9df12dc028af9e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal"
Sep 15 18:50:59.310583 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.310532    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f8d47b24220b0dba80a2c6cc3e11210a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-28.ec2.internal\" (UID: \"f8d47b24220b0dba80a2c6cc3e11210a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-28.ec2.internal"
Sep 15 18:50:59.310583 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.310521    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44b7a7d82c3ab405a9df12dc028af9e9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal\" (UID: \"44b7a7d82c3ab405a9df12dc028af9e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal"
Sep 15 18:50:59.383739 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:59.383663    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-28.ec2.internal\" not found"
Sep 15 18:50:59.459269 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.459246    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal"
Sep 15 18:50:59.462914 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.462896    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-28.ec2.internal"
Sep 15 18:50:59.484187 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:59.484167    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-28.ec2.internal\" not found"
Sep 15 18:50:59.585077 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:59.585028    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-28.ec2.internal\" not found"
Sep 15 18:50:59.687977 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:59.687909    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-28.ec2.internal\" not found"
Sep 15 18:50:59.767773 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.767742    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:59.789184 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:59.789155    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-28.ec2.internal\" not found"
Sep 15 18:50:59.842294 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.842264    2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 18:50:59.842806 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.842456    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 18:50:59.842806 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.842459    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 18:50:59.842806 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.842472    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 18:50:59.889449 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:50:59.889415    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-28.ec2.internal\" not found"
Sep 15 18:50:59.909160 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.909105    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 18:45:59 +0000 UTC" deadline="2028-04-11 09:48:05.859673409 +0000 UTC"
Sep 15 18:50:59.909160 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.909159    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13766h57m5.950518833s"
Sep 15 18:50:59.951412 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:50:59.951354    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:51:00.007956 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.007923    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal"
Sep 15 18:51:00.021393 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.021363    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 18:51:00.022507 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.022486    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-28.ec2.internal"
Sep 15 18:51:00.031006 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.030982    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 18:51:00.145823 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:00.145784    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b7a7d82c3ab405a9df12dc028af9e9.slice/crio-185c8ec4638ffeb089d87c3950d99cd59a19c69f46d58855892a41e3346dd858 WatchSource:0}: Error finding container 185c8ec4638ffeb089d87c3950d99cd59a19c69f46d58855892a41e3346dd858: Status 404 returned error can't find the container with id 185c8ec4638ffeb089d87c3950d99cd59a19c69f46d58855892a41e3346dd858
Sep 15 18:51:00.149357 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.149342    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 18:51:00.215656 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:00.215605    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d47b24220b0dba80a2c6cc3e11210a.slice/crio-add68bb02bea7aaad6fb762ab09b132b9a1d852d3d73a9c5c7f2aa427b6914bf WatchSource:0}: Error finding container add68bb02bea7aaad6fb762ab09b132b9a1d852d3d73a9c5c7f2aa427b6914bf: Status 404 returned error can't find the container with id add68bb02bea7aaad6fb762ab09b132b9a1d852d3d73a9c5c7f2aa427b6914bf
Sep 15 18:51:00.709148 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.709070    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:51:00.885750 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.885699    2555 apiserver.go:52] "Watching apiserver"
Sep 15 18:51:00.893191 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.893154    2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 18:51:00.893632 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.893604    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-xfzjp","openshift-multus/network-metrics-daemon-lpczh","openshift-network-operator/iptables-alerter-bf8kf","kube-system/global-pull-secret-syncer-j8kt6","kube-system/konnectivity-agent-tdwks","openshift-dns/node-resolver-d8r6c","openshift-multus/multus-additional-cni-plugins-2sq4v","openshift-network-diagnostics/network-check-target-wxhxs","openshift-ovn-kubernetes/ovnkube-node-cfcxz","kube-system/kube-apiserver-proxy-ip-10-0-132-28.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l","openshift-cluster-node-tuning-operator/tuned-b2mzg","openshift-image-registry/node-ca-g2fwg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal"]
Sep 15 18:51:00.897507 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.896676    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:00.899203 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.899174    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:00.899466 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.899443    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 18:51:00.900146 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.899847    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 18:51:00.900146 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.899900    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 18:51:00.900146 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.899941    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 18:51:00.900146 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.899975    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qp82p\""
Sep 15 18:51:00.900405 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.900344    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 18:51:00.901334 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.901193    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 18:51:00.901642 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.901622    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 18:51:00.901917 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.901896    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 18:51:00.902134 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.902112    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p248f\""
Sep 15 18:51:00.902530 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.902509    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.905046 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.905022    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 18:51:00.905145 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.905099    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:00.905893 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.905871    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 18:51:00.906341 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.906068    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n2zfj\""
Sep 15 18:51:00.906341 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.906209    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 18:51:00.906341 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.906290    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 18:51:00.906516 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.906380    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 18:51:00.906516 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.906438    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 18:51:00.906891 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.906839    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:00.906966 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:00.906922    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:00.907078 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.907062    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 18:51:00.907173 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.907156    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-h8txg\""
Sep 15 18:51:00.908262 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.907968    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 18:51:00.909799 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.909779    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:00.909886 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:00.909855    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:00.912292 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.912272    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d8r6c"
Sep 15 18:51:00.914259 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.914238    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 18:51:00.914590 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.914568    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-85sts\""
Sep 15 18:51:00.914953 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.914930    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tdwks"
Sep 15 18:51:00.915447 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.915424    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 18:51:00.916918 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.916894    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 18:51:00.919887 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.918838    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 18:51:00.919887 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.919279    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jzfs5\""
Sep 15 18:51:00.920115 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.920071    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bf8kf"
Sep 15 18:51:00.920659 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.920071    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-device-dir\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:00.920771 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.920725    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:00.920827 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.920765    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9777e14-6d89-422a-8c73-89744f2a4020-ovn-node-metrics-cert\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.920827 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.920802    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-cni-binary-copy\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:00.920930 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.920846    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-sys-fs\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:00.920930 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.920891    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-var-lib-openvswitch\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.921027 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.920929    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.921027 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.920961    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9777e14-6d89-422a-8c73-89744f2a4020-ovnkube-script-lib\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.921027 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.920994    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5cf60c84-6ebf-48cc-b094-f865d877bc03-multus-daemon-config\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:00.921173 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.921030    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-registration-dir\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:00.921173 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.921057    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-run-systemd\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.921173 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.921089    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-node-log\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.921173 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.921136    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-system-cni-dir\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:00.921358 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.921176    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-host-run-netns\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:00.921358 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.921207    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret\") pod \"global-pull-secret-syncer-j8kt6\" (UID: \"f16c1599-2406-48bc-88fb-36e9670dc851\") " pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:00.921358 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.921239    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-kubelet\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.921358 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.921271    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9777e14-6d89-422a-8c73-89744f2a4020-env-overrides\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.921358 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.921307    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-host-run-k8s-cni-cncf-io\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:00.921358 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.921342    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-host-var-lib-cni-multus\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:00.921634 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.921375    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-etc-kubernetes\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:00.921634 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.921405    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-system-cni-dir\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:00.921634 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.921437    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-etc-openvswitch\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.921634 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.921469    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c8j6\" (UniqueName: \"kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6\") pod \"network-check-target-wxhxs\" (UID: \"f9e0b82d-6132-4018-9b73-cb4bb6639ba2\") " pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:00.921634 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.921504    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htx9k\" (UniqueName: \"kubernetes.io/projected/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-kube-api-access-htx9k\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:00.921634 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.921542    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-multus-cni-dir\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:00.921634 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.921571    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rk2t\" (UniqueName: \"kubernetes.io/projected/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-kube-api-access-9rk2t\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:00.921634 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.921609    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-multus-socket-dir-parent\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:00.922013 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.921639    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-host-run-multus-certs\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:00.922824 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.922799    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 18:51:00.923133 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.923095    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv727\" (UniqueName: \"kubernetes.io/projected/5cf60c84-6ebf-48cc-b094-f865d877bc03-kube-api-access-mv727\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:00.923214 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.923155    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-host-var-lib-kubelet\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:00.923277 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.923226    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-multus-conf-dir\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:00.923339 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.923277    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:00.923510 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.923487    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-systemd-units\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.923635 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.923616    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 18:51:00.923752 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.923735    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j4xtd\""
Sep 15 18:51:00.923836 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.923809    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9777e14-6d89-422a-8c73-89744f2a4020-ovnkube-config\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.923902 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.923877    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 18:51:00.924185 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.924166    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:00.924264 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.923877    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbwdt\" (UniqueName: \"kubernetes.io/projected/f9777e14-6d89-422a-8c73-89744f2a4020-kube-api-access-rbwdt\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.924324 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.924294    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:51:00.924524 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.924498    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-run-openvswitch\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.924598 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.924577    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-socket-dir\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:00.924857 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.924619    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-cnibin\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:00.924930 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:00.924855    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:00.924930 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.924889    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:00.925033 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.924931    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-cnibin\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:00.925033 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.924971    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-slash\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.925141 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.925011    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-run-ovn\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.925141 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.925079    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-run-ovn-kubernetes\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.925242 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.925148    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cf60c84-6ebf-48cc-b094-f865d877bc03-cni-binary-copy\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:00.925242 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.925185    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f16c1599-2406-48bc-88fb-36e9670dc851-kubelet-config\") pod \"global-pull-secret-syncer-j8kt6\" (UID: \"f16c1599-2406-48bc-88fb-36e9670dc851\") " pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:00.925350 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.925248    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-run-netns\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.925350 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.925289    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-cni-netd\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.925350 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.925326    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:00.925499 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.925395    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-etc-selinux\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:00.925499 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.925458    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-log-socket\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.925499 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.925494    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-host-var-lib-cni-bin\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:00.925641 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.925553    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-hostroot\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:00.925827 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.925791    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f16c1599-2406-48bc-88fb-36e9670dc851-dbus\") pod \"global-pull-secret-syncer-j8kt6\" (UID: \"f16c1599-2406-48bc-88fb-36e9670dc851\") " pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:00.925889 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.925856    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-cni-bin\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:00.928202 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.928181    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:00.930431 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.930270    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 18:51:00.930431 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.930286    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:51:00.930431 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.930314    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2nvrr\""
Sep 15 18:51:00.930732 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.930700    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g2fwg"
Sep 15 18:51:00.932977 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.932958    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 18:51:00.933128 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.933114    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 18:51:00.933302 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.933286    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 18:51:00.933425 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.933413    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rdntb\""
Sep 15 18:51:00.942561 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.942537    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dkw74"
Sep 15 18:51:00.949970 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:00.949913    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dkw74"
Sep 15 18:51:01.009138 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.009102    2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 18:51:01.026839 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.026808    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htx9k\" (UniqueName: \"kubernetes.io/projected/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-kube-api-access-htx9k\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:01.027001 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.026854    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-multus-cni-dir\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.027001 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.026889    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-systemd\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.027001 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.026918    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rk2t\" (UniqueName: \"kubernetes.io/projected/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-kube-api-access-9rk2t\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:01.027001 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.026946    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-multus-socket-dir-parent\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.027001 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.026973    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-host-run-multus-certs\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.027001 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.026997    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv727\" (UniqueName: \"kubernetes.io/projected/5cf60c84-6ebf-48cc-b094-f865d877bc03-kube-api-access-mv727\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.027366 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027023    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-run\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.027366 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027055    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-host\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.027366 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027078    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-tuned\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.027366 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027106    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-966h7\" (UniqueName: \"kubernetes.io/projected/4824dade-a75e-4ddc-b086-989eaedafe12-kube-api-access-966h7\") pod \"node-ca-g2fwg\" (UID: \"4824dade-a75e-4ddc-b086-989eaedafe12\") " pod="openshift-image-registry/node-ca-g2fwg"
Sep 15 18:51:01.027366 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027134    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-host-var-lib-kubelet\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.027366 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027158    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-multus-conf-dir\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.027366 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027184    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:01.027366 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027210    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-systemd-units\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.027366 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027244    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9777e14-6d89-422a-8c73-89744f2a4020-ovnkube-config\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.027366 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027270    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbwdt\" (UniqueName: \"kubernetes.io/projected/f9777e14-6d89-422a-8c73-89744f2a4020-kube-api-access-rbwdt\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.027366 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027294    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-sysconfig\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.027366 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027318    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-sysctl-conf\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.027366 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027346    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-run-openvswitch\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.028164 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027378    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-socket-dir\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:01.028164 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027404    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-cnibin\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:01.028164 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027430    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:01.028164 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027462    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-cnibin\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.028164 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027487    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-var-lib-kubelet\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.028164 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027529    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-slash\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.028164 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027595    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-slash\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.028164 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027595    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-host-var-lib-kubelet\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.028164 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027599    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-run-ovn\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.028164 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027648    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-run-openvswitch\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.028164 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027654    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-run-ovn-kubernetes\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.028164 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027647    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-run-ovn\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.028164 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027684    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cf60c84-6ebf-48cc-b094-f865d877bc03-cni-binary-copy\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.028164 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027723    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-multus-conf-dir\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.028164 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027763    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-systemd-units\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.028164 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027810    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f16c1599-2406-48bc-88fb-36e9670dc851-kubelet-config\") pod \"global-pull-secret-syncer-j8kt6\" (UID: \"f16c1599-2406-48bc-88fb-36e9670dc851\") " pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:01.028164 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027853    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-cnibin\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:01.028164 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.028051    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-multus-cni-dir\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.029119 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.028298    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9777e14-6d89-422a-8c73-89744f2a4020-ovnkube-config\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.029119 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.028342    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-multus-socket-dir-parent\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.029119 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.028419    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-host-run-multus-certs\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.029119 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.028436    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-socket-dir\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:01.029119 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.028461    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:01.029119 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.028525    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-cnibin\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.029119 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.028722    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-run-ovn-kubernetes\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.029119 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.027726    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f16c1599-2406-48bc-88fb-36e9670dc851-kubelet-config\") pod \"global-pull-secret-syncer-j8kt6\" (UID: \"f16c1599-2406-48bc-88fb-36e9670dc851\") " pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:01.029119 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.028770    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-sysctl-d\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.029119 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.028795    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4824dade-a75e-4ddc-b086-989eaedafe12-serviceca\") pod \"node-ca-g2fwg\" (UID: \"4824dade-a75e-4ddc-b086-989eaedafe12\") " pod="openshift-image-registry/node-ca-g2fwg"
Sep 15 18:51:01.029119 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.028838    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-run-netns\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.029119 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.028864    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-cni-netd\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.029119 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.028892    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:01.029119 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.028917    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-etc-selinux\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:01.029119 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.028940    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-log-socket\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.029119 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.028965    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-host-var-lib-cni-bin\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.029119 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.028988    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-hostroot\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.029981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029012    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f16c1599-2406-48bc-88fb-36e9670dc851-dbus\") pod \"global-pull-secret-syncer-j8kt6\" (UID: \"f16c1599-2406-48bc-88fb-36e9670dc851\") " pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:01.029981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029041    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9555dd5-d766-4b43-bd0b-e2ed73acc9cd-tmp-dir\") pod \"node-resolver-d8r6c\" (UID: \"c9555dd5-d766-4b43-bd0b-e2ed73acc9cd\") " pod="openshift-dns/node-resolver-d8r6c"
Sep 15 18:51:01.029981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029064    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-sys\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.029981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029093    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-cni-bin\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.029981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029119    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd565903-2276-469c-a07c-63cd015ef00a-konnectivity-ca\") pod \"konnectivity-agent-tdwks\" (UID: \"cd565903-2276-469c-a07c-63cd015ef00a\") " pod="kube-system/konnectivity-agent-tdwks"
Sep 15 18:51:01.029981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029156    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbqnj\" (UniqueName: \"kubernetes.io/projected/0a36af7f-abf7-414c-80b2-6314083e1a60-kube-api-access-hbqnj\") pod \"network-metrics-daemon-lpczh\" (UID: \"0a36af7f-abf7-414c-80b2-6314083e1a60\") " pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:01.029981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029190    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-kubernetes\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.029981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029215    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-device-dir\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:01.029981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029233    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:01.029981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029250    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9777e14-6d89-422a-8c73-89744f2a4020-ovn-node-metrics-cert\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.029981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029276    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-cni-binary-copy\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:01.029981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029290    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cf60c84-6ebf-48cc-b094-f865d877bc03-cni-binary-copy\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.029981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029302    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-sys-fs\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:01.029981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029330    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-var-lib-openvswitch\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.029981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029361    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.029981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029376    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f16c1599-2406-48bc-88fb-36e9670dc851-dbus\") pod \"global-pull-secret-syncer-j8kt6\" (UID: \"f16c1599-2406-48bc-88fb-36e9670dc851\") " pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:01.029981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029390    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9777e14-6d89-422a-8c73-89744f2a4020-ovnkube-script-lib\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.030954 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029427    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5cf60c84-6ebf-48cc-b094-f865d877bc03-multus-daemon-config\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.030954 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029510    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-etc-selinux\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:01.030954 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029540    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzq4n\" (UniqueName: \"kubernetes.io/projected/5de8260e-f158-4eca-a056-8e5e039c45cc-kube-api-access-pzq4n\") pod \"iptables-alerter-bf8kf\" (UID: \"5de8260e-f158-4eca-a056-8e5e039c45cc\") " pod="openshift-network-operator/iptables-alerter-bf8kf"
Sep 15 18:51:01.030954 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029567    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-run-netns\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.030954 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029574    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-tmp\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.030954 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029608    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-registration-dir\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:01.030954 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029617    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:01.030954 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029638    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-run-systemd\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.030954 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029609    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-cni-netd\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.030954 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029667    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:01.030954 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029692    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-host-var-lib-cni-bin\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.030954 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029707    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-log-socket\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.030954 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029745    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-cni-bin\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.030954 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029821    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-hostroot\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.030954 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029834    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-device-dir\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:01.030954 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029881    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-var-lib-openvswitch\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.030954 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029913    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9777e14-6d89-422a-8c73-89744f2a4020-ovnkube-script-lib\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.031889 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029937    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-sys-fs\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:01.031889 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.029974    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-registration-dir\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:01.031889 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030021    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:01.031889 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030089    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-run-systemd\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.031889 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030137    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.031889 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030225    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-cni-binary-copy\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:01.031889 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030264    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5cf60c84-6ebf-48cc-b094-f865d877bc03-multus-daemon-config\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.031889 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030302    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-node-log\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.031889 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030334    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-system-cni-dir\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.031889 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030356    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-node-log\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.031889 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030362    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-host-run-netns\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.031889 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030388    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9555dd5-d766-4b43-bd0b-e2ed73acc9cd-hosts-file\") pod \"node-resolver-d8r6c\" (UID: \"c9555dd5-d766-4b43-bd0b-e2ed73acc9cd\") " pod="openshift-dns/node-resolver-d8r6c"
Sep 15 18:51:01.031889 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030412    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqcfr\" (UniqueName: \"kubernetes.io/projected/c9555dd5-d766-4b43-bd0b-e2ed73acc9cd-kube-api-access-nqcfr\") pod \"node-resolver-d8r6c\" (UID: \"c9555dd5-d766-4b43-bd0b-e2ed73acc9cd\") " pod="openshift-dns/node-resolver-d8r6c"
Sep 15 18:51:01.031889 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030437    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs\") pod \"network-metrics-daemon-lpczh\" (UID: \"0a36af7f-abf7-414c-80b2-6314083e1a60\") " pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:01.031889 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030464    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret\") pod \"global-pull-secret-syncer-j8kt6\" (UID: \"f16c1599-2406-48bc-88fb-36e9670dc851\") " pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:01.031889 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030489    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-modprobe-d\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.031889 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030515    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-kubelet\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.032841 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030531    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-host-run-netns\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.032841 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030542    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9777e14-6d89-422a-8c73-89744f2a4020-env-overrides\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.032841 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030492    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-system-cni-dir\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.032841 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030602    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-host-run-k8s-cni-cncf-io\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.032841 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:01.030637    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:01.032841 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030638    2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 18:51:01.032841 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030702    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-host-kubelet\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.032841 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030733    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-host-var-lib-cni-multus\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.032841 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:01.030781    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret podName:f16c1599-2406-48bc-88fb-36e9670dc851 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:01.530752508 +0000 UTC m=+3.018081494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret") pod "global-pull-secret-syncer-j8kt6" (UID: "f16c1599-2406-48bc-88fb-36e9670dc851") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:01.032841 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030800    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-etc-kubernetes\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.032841 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.030914    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-host-run-k8s-cni-cncf-io\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.032841 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.031122    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9777e14-6d89-422a-8c73-89744f2a4020-env-overrides\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.032841 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.031137    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd565903-2276-469c-a07c-63cd015ef00a-agent-certs\") pod \"konnectivity-agent-tdwks\" (UID: \"cd565903-2276-469c-a07c-63cd015ef00a\") " pod="kube-system/konnectivity-agent-tdwks"
Sep 15 18:51:01.032841 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.031157    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-etc-kubernetes\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.032841 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.031171    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5cf60c84-6ebf-48cc-b094-f865d877bc03-host-var-lib-cni-multus\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.032841 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.031210    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5de8260e-f158-4eca-a056-8e5e039c45cc-iptables-alerter-script\") pod \"iptables-alerter-bf8kf\" (UID: \"5de8260e-f158-4eca-a056-8e5e039c45cc\") " pod="openshift-network-operator/iptables-alerter-bf8kf"
Sep 15 18:51:01.032841 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.031238    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdsgd\" (UniqueName: \"kubernetes.io/projected/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-kube-api-access-qdsgd\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.033776 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.031277    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-system-cni-dir\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:01.033776 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.031300    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-etc-openvswitch\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.033776 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.031322    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8j6\" (UniqueName: \"kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6\") pod \"network-check-target-wxhxs\" (UID: \"f9e0b82d-6132-4018-9b73-cb4bb6639ba2\") " pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:01.033776 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.031352    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5de8260e-f158-4eca-a056-8e5e039c45cc-host-slash\") pod \"iptables-alerter-bf8kf\" (UID: \"5de8260e-f158-4eca-a056-8e5e039c45cc\") " pod="openshift-network-operator/iptables-alerter-bf8kf"
Sep 15 18:51:01.033776 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.031382    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-system-cni-dir\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:01.033776 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.031421    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-lib-modules\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.033776 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.031458    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9777e14-6d89-422a-8c73-89744f2a4020-etc-openvswitch\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.033776 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.031502    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4824dade-a75e-4ddc-b086-989eaedafe12-host\") pod \"node-ca-g2fwg\" (UID: \"4824dade-a75e-4ddc-b086-989eaedafe12\") " pod="openshift-image-registry/node-ca-g2fwg"
Sep 15 18:51:01.038565 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.037224    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htx9k\" (UniqueName: \"kubernetes.io/projected/2077960a-71c3-4fb4-8220-9fc0bf5f40e0-kube-api-access-htx9k\") pod \"aws-ebs-csi-driver-node-fkj5l\" (UID: \"2077960a-71c3-4fb4-8220-9fc0bf5f40e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:01.039210 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.039114    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9777e14-6d89-422a-8c73-89744f2a4020-ovn-node-metrics-cert\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.039787 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:01.039764    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:51:01.039896 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:01.039828    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:51:01.039896 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:01.039865    2555 projected.go:196] Error preparing data for projected volume kube-api-access-2c8j6 for pod openshift-network-diagnostics/network-check-target-wxhxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:01.040024 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:01.039948    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6 podName:f9e0b82d-6132-4018-9b73-cb4bb6639ba2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:01.539923955 +0000 UTC m=+3.027252920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2c8j6" (UniqueName: "kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6") pod "network-check-target-wxhxs" (UID: "f9e0b82d-6132-4018-9b73-cb4bb6639ba2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:01.042310 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.042280    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv727\" (UniqueName: \"kubernetes.io/projected/5cf60c84-6ebf-48cc-b094-f865d877bc03-kube-api-access-mv727\") pod \"multus-xfzjp\" (UID: \"5cf60c84-6ebf-48cc-b094-f865d877bc03\") " pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.043702 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.043620    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-28.ec2.internal" event={"ID":"f8d47b24220b0dba80a2c6cc3e11210a","Type":"ContainerStarted","Data":"add68bb02bea7aaad6fb762ab09b132b9a1d852d3d73a9c5c7f2aa427b6914bf"}
Sep 15 18:51:01.045637 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.045609    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal" event={"ID":"44b7a7d82c3ab405a9df12dc028af9e9","Type":"ContainerStarted","Data":"185c8ec4638ffeb089d87c3950d99cd59a19c69f46d58855892a41e3346dd858"}
Sep 15 18:51:01.047266 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.047241    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbwdt\" (UniqueName: \"kubernetes.io/projected/f9777e14-6d89-422a-8c73-89744f2a4020-kube-api-access-rbwdt\") pod \"ovnkube-node-cfcxz\" (UID: \"f9777e14-6d89-422a-8c73-89744f2a4020\") " pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.047416 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.047385    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rk2t\" (UniqueName: \"kubernetes.io/projected/598c3d78-c05f-42dc-ac4f-0a6640c6fc38-kube-api-access-9rk2t\") pod \"multus-additional-cni-plugins-2sq4v\" (UID: \"598c3d78-c05f-42dc-ac4f-0a6640c6fc38\") " pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:01.085798 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.085765    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:51:01.132358 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132323    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-sysconfig\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.132558 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132368    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-sysctl-conf\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.132558 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132396    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-var-lib-kubelet\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.132558 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132421    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-sysctl-d\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.132558 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132446    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4824dade-a75e-4ddc-b086-989eaedafe12-serviceca\") pod \"node-ca-g2fwg\" (UID: \"4824dade-a75e-4ddc-b086-989eaedafe12\") " pod="openshift-image-registry/node-ca-g2fwg"
Sep 15 18:51:01.132558 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132475    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9555dd5-d766-4b43-bd0b-e2ed73acc9cd-tmp-dir\") pod \"node-resolver-d8r6c\" (UID: \"c9555dd5-d766-4b43-bd0b-e2ed73acc9cd\") " pod="openshift-dns/node-resolver-d8r6c"
Sep 15 18:51:01.132558 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132498    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-sys\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.132558 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132509    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-var-lib-kubelet\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.132558 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132525    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-sysconfig\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.132558 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132520    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd565903-2276-469c-a07c-63cd015ef00a-konnectivity-ca\") pod \"konnectivity-agent-tdwks\" (UID: \"cd565903-2276-469c-a07c-63cd015ef00a\") " pod="kube-system/konnectivity-agent-tdwks"
Sep 15 18:51:01.133035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132582    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbqnj\" (UniqueName: \"kubernetes.io/projected/0a36af7f-abf7-414c-80b2-6314083e1a60-kube-api-access-hbqnj\") pod \"network-metrics-daemon-lpczh\" (UID: \"0a36af7f-abf7-414c-80b2-6314083e1a60\") " pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:01.133035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132610    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-kubernetes\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.133035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132638    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-sys\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.133035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132665    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzq4n\" (UniqueName: \"kubernetes.io/projected/5de8260e-f158-4eca-a056-8e5e039c45cc-kube-api-access-pzq4n\") pod \"iptables-alerter-bf8kf\" (UID: \"5de8260e-f158-4eca-a056-8e5e039c45cc\") " pod="openshift-network-operator/iptables-alerter-bf8kf"
Sep 15 18:51:01.133035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132580    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-sysctl-conf\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.133035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132660    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-sysctl-d\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.133035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132707    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-tmp\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.133035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132728    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-kubernetes\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.133035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132741    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9555dd5-d766-4b43-bd0b-e2ed73acc9cd-hosts-file\") pod \"node-resolver-d8r6c\" (UID: \"c9555dd5-d766-4b43-bd0b-e2ed73acc9cd\") " pod="openshift-dns/node-resolver-d8r6c"
Sep 15 18:51:01.133035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132771    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqcfr\" (UniqueName: \"kubernetes.io/projected/c9555dd5-d766-4b43-bd0b-e2ed73acc9cd-kube-api-access-nqcfr\") pod \"node-resolver-d8r6c\" (UID: \"c9555dd5-d766-4b43-bd0b-e2ed73acc9cd\") " pod="openshift-dns/node-resolver-d8r6c"
Sep 15 18:51:01.133035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132797    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs\") pod \"network-metrics-daemon-lpczh\" (UID: \"0a36af7f-abf7-414c-80b2-6314083e1a60\") " pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:01.133035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132836    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-modprobe-d\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.133035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132864    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd565903-2276-469c-a07c-63cd015ef00a-agent-certs\") pod \"konnectivity-agent-tdwks\" (UID: \"cd565903-2276-469c-a07c-63cd015ef00a\") " pod="kube-system/konnectivity-agent-tdwks"
Sep 15 18:51:01.133035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132889    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5de8260e-f158-4eca-a056-8e5e039c45cc-iptables-alerter-script\") pod \"iptables-alerter-bf8kf\" (UID: \"5de8260e-f158-4eca-a056-8e5e039c45cc\") " pod="openshift-network-operator/iptables-alerter-bf8kf"
Sep 15 18:51:01.133035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132914    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdsgd\" (UniqueName: \"kubernetes.io/projected/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-kube-api-access-qdsgd\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.133035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.132974    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5de8260e-f158-4eca-a056-8e5e039c45cc-host-slash\") pod \"iptables-alerter-bf8kf\" (UID: \"5de8260e-f158-4eca-a056-8e5e039c45cc\") " pod="openshift-network-operator/iptables-alerter-bf8kf"
Sep 15 18:51:01.133035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.133002    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-lib-modules\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.133890 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.133026    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4824dade-a75e-4ddc-b086-989eaedafe12-host\") pod \"node-ca-g2fwg\" (UID: \"4824dade-a75e-4ddc-b086-989eaedafe12\") " pod="openshift-image-registry/node-ca-g2fwg"
Sep 15 18:51:01.133890 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.133036    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9555dd5-d766-4b43-bd0b-e2ed73acc9cd-hosts-file\") pod \"node-resolver-d8r6c\" (UID: \"c9555dd5-d766-4b43-bd0b-e2ed73acc9cd\") " pod="openshift-dns/node-resolver-d8r6c"
Sep 15 18:51:01.133890 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.133052    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-systemd\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.133890 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.133081    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-run\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.133890 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.133104    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-host\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.133890 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.133128    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-tuned\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.133890 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.133131    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4824dade-a75e-4ddc-b086-989eaedafe12-serviceca\") pod \"node-ca-g2fwg\" (UID: \"4824dade-a75e-4ddc-b086-989eaedafe12\") " pod="openshift-image-registry/node-ca-g2fwg"
Sep 15 18:51:01.133890 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.133153    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-966h7\" (UniqueName: \"kubernetes.io/projected/4824dade-a75e-4ddc-b086-989eaedafe12-kube-api-access-966h7\") pod \"node-ca-g2fwg\" (UID: \"4824dade-a75e-4ddc-b086-989eaedafe12\") " pod="openshift-image-registry/node-ca-g2fwg"
Sep 15 18:51:01.133890 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.133205    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd565903-2276-469c-a07c-63cd015ef00a-konnectivity-ca\") pod \"konnectivity-agent-tdwks\" (UID: \"cd565903-2276-469c-a07c-63cd015ef00a\") " pod="kube-system/konnectivity-agent-tdwks"
Sep 15 18:51:01.133890 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:01.133305    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:01.133890 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.133339    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-lib-modules\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.133890 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:01.133368    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs podName:0a36af7f-abf7-414c-80b2-6314083e1a60 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:01.633342233 +0000 UTC m=+3.120671215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs") pod "network-metrics-daemon-lpczh" (UID: "0a36af7f-abf7-414c-80b2-6314083e1a60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:01.133890 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.133388    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5de8260e-f158-4eca-a056-8e5e039c45cc-host-slash\") pod \"iptables-alerter-bf8kf\" (UID: \"5de8260e-f158-4eca-a056-8e5e039c45cc\") " pod="openshift-network-operator/iptables-alerter-bf8kf"
Sep 15 18:51:01.133890 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.133473    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-modprobe-d\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.133890 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.133479    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4824dade-a75e-4ddc-b086-989eaedafe12-host\") pod \"node-ca-g2fwg\" (UID: \"4824dade-a75e-4ddc-b086-989eaedafe12\") " pod="openshift-image-registry/node-ca-g2fwg"
Sep 15 18:51:01.133890 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.133539    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-systemd\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.133890 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.133570    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-host\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.133890 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.133624    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-run\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.134845 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.134043    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5de8260e-f158-4eca-a056-8e5e039c45cc-iptables-alerter-script\") pod \"iptables-alerter-bf8kf\" (UID: \"5de8260e-f158-4eca-a056-8e5e039c45cc\") " pod="openshift-network-operator/iptables-alerter-bf8kf"
Sep 15 18:51:01.134845 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.134292    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9555dd5-d766-4b43-bd0b-e2ed73acc9cd-tmp-dir\") pod \"node-resolver-d8r6c\" (UID: \"c9555dd5-d766-4b43-bd0b-e2ed73acc9cd\") " pod="openshift-dns/node-resolver-d8r6c"
Sep 15 18:51:01.135687 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.135642    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-tmp\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.136219 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.136171    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd565903-2276-469c-a07c-63cd015ef00a-agent-certs\") pod \"konnectivity-agent-tdwks\" (UID: \"cd565903-2276-469c-a07c-63cd015ef00a\") " pod="kube-system/konnectivity-agent-tdwks"
Sep 15 18:51:01.136527 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.136505    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-etc-tuned\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.142597 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.142569    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-966h7\" (UniqueName: \"kubernetes.io/projected/4824dade-a75e-4ddc-b086-989eaedafe12-kube-api-access-966h7\") pod \"node-ca-g2fwg\" (UID: \"4824dade-a75e-4ddc-b086-989eaedafe12\") " pod="openshift-image-registry/node-ca-g2fwg"
Sep 15 18:51:01.144327 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.144303    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzq4n\" (UniqueName: \"kubernetes.io/projected/5de8260e-f158-4eca-a056-8e5e039c45cc-kube-api-access-pzq4n\") pod \"iptables-alerter-bf8kf\" (UID: \"5de8260e-f158-4eca-a056-8e5e039c45cc\") " pod="openshift-network-operator/iptables-alerter-bf8kf"
Sep 15 18:51:01.144416 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.144324    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqcfr\" (UniqueName: \"kubernetes.io/projected/c9555dd5-d766-4b43-bd0b-e2ed73acc9cd-kube-api-access-nqcfr\") pod \"node-resolver-d8r6c\" (UID: \"c9555dd5-d766-4b43-bd0b-e2ed73acc9cd\") " pod="openshift-dns/node-resolver-d8r6c"
Sep 15 18:51:01.147119 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.147066    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbqnj\" (UniqueName: \"kubernetes.io/projected/0a36af7f-abf7-414c-80b2-6314083e1a60-kube-api-access-hbqnj\") pod \"network-metrics-daemon-lpczh\" (UID: \"0a36af7f-abf7-414c-80b2-6314083e1a60\") " pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:01.147423 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.147360    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdsgd\" (UniqueName: \"kubernetes.io/projected/fc83f4f3-e01c-4e16-86cd-8eadc1ad6307-kube-api-access-qdsgd\") pod \"tuned-b2mzg\" (UID: \"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307\") " pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.212567 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.212485    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2sq4v"
Sep 15 18:51:01.226044 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.226013    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l"
Sep 15 18:51:01.239123 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.239097    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:01.246982 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.246954    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xfzjp"
Sep 15 18:51:01.255789 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.255761    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d8r6c"
Sep 15 18:51:01.270816 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.270790    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tdwks"
Sep 15 18:51:01.281733 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.281708    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bf8kf"
Sep 15 18:51:01.289663 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.289641    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b2mzg"
Sep 15 18:51:01.296772 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.296748    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g2fwg"
Sep 15 18:51:01.537272 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.537205    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret\") pod \"global-pull-secret-syncer-j8kt6\" (UID: \"f16c1599-2406-48bc-88fb-36e9670dc851\") " pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:01.537403 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:01.537327    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:01.537403 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:01.537385    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret podName:f16c1599-2406-48bc-88fb-36e9670dc851 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:02.537369547 +0000 UTC m=+4.024698512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret") pod "global-pull-secret-syncer-j8kt6" (UID: "f16c1599-2406-48bc-88fb-36e9670dc851") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:01.637515 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.637477    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs\") pod \"network-metrics-daemon-lpczh\" (UID: \"0a36af7f-abf7-414c-80b2-6314083e1a60\") " pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:01.637707 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.637545    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8j6\" (UniqueName: \"kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6\") pod \"network-check-target-wxhxs\" (UID: \"f9e0b82d-6132-4018-9b73-cb4bb6639ba2\") " pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:01.637707 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:01.637607    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:01.637707 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:01.637689    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs podName:0a36af7f-abf7-414c-80b2-6314083e1a60 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:02.637667359 +0000 UTC m=+4.124996341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs") pod "network-metrics-daemon-lpczh" (UID: "0a36af7f-abf7-414c-80b2-6314083e1a60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:01.637896 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:01.637720    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:51:01.637896 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:01.637736    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:51:01.637896 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:01.637748    2555 projected.go:196] Error preparing data for projected volume kube-api-access-2c8j6 for pod openshift-network-diagnostics/network-check-target-wxhxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:01.637896 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:01.637793    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6 podName:f9e0b82d-6132-4018-9b73-cb4bb6639ba2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:02.637780244 +0000 UTC m=+4.125109219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2c8j6" (UniqueName: "kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6") pod "network-check-target-wxhxs" (UID: "f9e0b82d-6132-4018-9b73-cb4bb6639ba2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:01.829243 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.829152    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:51:01.953910 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.953842    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 18:46:01 +0000 UTC" deadline="2028-03-05 11:42:30.931585842 +0000 UTC"
Sep 15 18:51:01.953910 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:01.953886    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12880h51m28.977705885s"
Sep 15 18:51:02.281449 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:02.281208    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd565903_2276_469c_a07c_63cd015ef00a.slice/crio-92931bbd5d70a82f987d2abd7ae817940e4202a00e84e9da8d14403241bcfd92 WatchSource:0}: Error finding container 92931bbd5d70a82f987d2abd7ae817940e4202a00e84e9da8d14403241bcfd92: Status 404 returned error can't find the container with id 92931bbd5d70a82f987d2abd7ae817940e4202a00e84e9da8d14403241bcfd92
Sep 15 18:51:02.303932 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:02.303906    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc83f4f3_e01c_4e16_86cd_8eadc1ad6307.slice/crio-a0a6b25eee188e08d940bb0764ef473d95d17201263f183621eef62b63c36c04 WatchSource:0}: Error finding container a0a6b25eee188e08d940bb0764ef473d95d17201263f183621eef62b63c36c04: Status 404 returned error can't find the container with id a0a6b25eee188e08d940bb0764ef473d95d17201263f183621eef62b63c36c04
Sep 15 18:51:02.308363 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:02.308337    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9555dd5_d766_4b43_bd0b_e2ed73acc9cd.slice/crio-d3afe795033fdfba74ad15a87320cdaca62821660d4505838bfc152c13e24c1f WatchSource:0}: Error finding container d3afe795033fdfba74ad15a87320cdaca62821660d4505838bfc152c13e24c1f: Status 404 returned error can't find the container with id d3afe795033fdfba74ad15a87320cdaca62821660d4505838bfc152c13e24c1f
Sep 15 18:51:02.314734 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:02.314708    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9777e14_6d89_422a_8c73_89744f2a4020.slice/crio-7077244b0a3ceb6fa9f3f184cf2c1363844c98ab0d0a1028776a9b973ffe4f02 WatchSource:0}: Error finding container 7077244b0a3ceb6fa9f3f184cf2c1363844c98ab0d0a1028776a9b973ffe4f02: Status 404 returned error can't find the container with id 7077244b0a3ceb6fa9f3f184cf2c1363844c98ab0d0a1028776a9b973ffe4f02
Sep 15 18:51:02.315947 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:02.315928    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de8260e_f158_4eca_a056_8e5e039c45cc.slice/crio-c2e8a38937d65ddf12a755fc1bb90ced27a5a6b9f6939201cb0b203b928336ed WatchSource:0}: Error finding container c2e8a38937d65ddf12a755fc1bb90ced27a5a6b9f6939201cb0b203b928336ed: Status 404 returned error can't find the container with id c2e8a38937d65ddf12a755fc1bb90ced27a5a6b9f6939201cb0b203b928336ed
Sep 15 18:51:02.317527 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:02.317460    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2077960a_71c3_4fb4_8220_9fc0bf5f40e0.slice/crio-897ff0f321a7d2b4a528659e9595dc077c1695051382b4c159bee8ff4ec7fe81 WatchSource:0}: Error finding container 897ff0f321a7d2b4a528659e9595dc077c1695051382b4c159bee8ff4ec7fe81: Status 404 returned error can't find the container with id 897ff0f321a7d2b4a528659e9595dc077c1695051382b4c159bee8ff4ec7fe81
Sep 15 18:51:02.319005 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:02.318592    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf60c84_6ebf_48cc_b094_f865d877bc03.slice/crio-9f690613ffcfd181e02db5435fd4708433efd5ea7c7e5eb524b779c8ecf90f2d WatchSource:0}: Error finding container 9f690613ffcfd181e02db5435fd4708433efd5ea7c7e5eb524b779c8ecf90f2d: Status 404 returned error can't find the container with id 9f690613ffcfd181e02db5435fd4708433efd5ea7c7e5eb524b779c8ecf90f2d
Sep 15 18:51:02.319675 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:02.319651    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4824dade_a75e_4ddc_b086_989eaedafe12.slice/crio-e7f373c9dc17e05a8540a6470a85fd463048fabdfde9a40eba5d80792ae3707d WatchSource:0}: Error finding container e7f373c9dc17e05a8540a6470a85fd463048fabdfde9a40eba5d80792ae3707d: Status 404 returned error can't find the container with id e7f373c9dc17e05a8540a6470a85fd463048fabdfde9a40eba5d80792ae3707d
Sep 15 18:51:02.321169 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:02.320201    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598c3d78_c05f_42dc_ac4f_0a6640c6fc38.slice/crio-85ecf91a075e9cd07d5f5204dcde7b1e722445a78a1f0e3cd739d146e9cfc786 WatchSource:0}: Error finding container 85ecf91a075e9cd07d5f5204dcde7b1e722445a78a1f0e3cd739d146e9cfc786: Status 404 returned error can't find the container with id 85ecf91a075e9cd07d5f5204dcde7b1e722445a78a1f0e3cd739d146e9cfc786
Sep 15 18:51:02.544717 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:02.544619    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret\") pod \"global-pull-secret-syncer-j8kt6\" (UID: \"f16c1599-2406-48bc-88fb-36e9670dc851\") " pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:02.544892 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:02.544776    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:02.544892 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:02.544856    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret podName:f16c1599-2406-48bc-88fb-36e9670dc851 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:04.544835655 +0000 UTC m=+6.032164620 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret") pod "global-pull-secret-syncer-j8kt6" (UID: "f16c1599-2406-48bc-88fb-36e9670dc851") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:02.645503 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:02.645476    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs\") pod \"network-metrics-daemon-lpczh\" (UID: \"0a36af7f-abf7-414c-80b2-6314083e1a60\") " pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:02.645620 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:02.645539    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8j6\" (UniqueName: \"kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6\") pod \"network-check-target-wxhxs\" (UID: \"f9e0b82d-6132-4018-9b73-cb4bb6639ba2\") " pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:02.645698 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:02.645674    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:02.645751 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:02.645699    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:51:02.645802 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:02.645756    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:51:02.645802 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:02.645760    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs podName:0a36af7f-abf7-414c-80b2-6314083e1a60 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:04.645734193 +0000 UTC m=+6.133063163 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs") pod "network-metrics-daemon-lpczh" (UID: "0a36af7f-abf7-414c-80b2-6314083e1a60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:02.645802 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:02.645772    2555 projected.go:196] Error preparing data for projected volume kube-api-access-2c8j6 for pod openshift-network-diagnostics/network-check-target-wxhxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:02.645945 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:02.645826    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6 podName:f9e0b82d-6132-4018-9b73-cb4bb6639ba2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:04.645807919 +0000 UTC m=+6.133136904 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2c8j6" (UniqueName: "kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6") pod "network-check-target-wxhxs" (UID: "f9e0b82d-6132-4018-9b73-cb4bb6639ba2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:02.955031 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:02.954863    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 18:46:01 +0000 UTC" deadline="2028-03-29 03:35:12.161524057 +0000 UTC"
Sep 15 18:51:02.955031 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:02.954942    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13448h44m9.206586761s"
Sep 15 18:51:03.038017 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:03.036631    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:03.038017 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:03.036788    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:03.038017 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:03.036884    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:03.038017 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:03.036996    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:03.038017 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:03.037074    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:03.038017 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:03.037166    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:03.051844 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:03.051619    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz" event={"ID":"f9777e14-6d89-422a-8c73-89744f2a4020","Type":"ContainerStarted","Data":"7077244b0a3ceb6fa9f3f184cf2c1363844c98ab0d0a1028776a9b973ffe4f02"}
Sep 15 18:51:03.056296 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:03.056204    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sq4v" event={"ID":"598c3d78-c05f-42dc-ac4f-0a6640c6fc38","Type":"ContainerStarted","Data":"85ecf91a075e9cd07d5f5204dcde7b1e722445a78a1f0e3cd739d146e9cfc786"}
Sep 15 18:51:03.062192 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:03.062105    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g2fwg" event={"ID":"4824dade-a75e-4ddc-b086-989eaedafe12","Type":"ContainerStarted","Data":"e7f373c9dc17e05a8540a6470a85fd463048fabdfde9a40eba5d80792ae3707d"}
Sep 15 18:51:03.065976 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:03.065673    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d8r6c" event={"ID":"c9555dd5-d766-4b43-bd0b-e2ed73acc9cd","Type":"ContainerStarted","Data":"d3afe795033fdfba74ad15a87320cdaca62821660d4505838bfc152c13e24c1f"}
Sep 15 18:51:03.071873 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:03.071801    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b2mzg" event={"ID":"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307","Type":"ContainerStarted","Data":"a0a6b25eee188e08d940bb0764ef473d95d17201263f183621eef62b63c36c04"}
Sep 15 18:51:03.074360 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:03.074293    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tdwks" event={"ID":"cd565903-2276-469c-a07c-63cd015ef00a","Type":"ContainerStarted","Data":"92931bbd5d70a82f987d2abd7ae817940e4202a00e84e9da8d14403241bcfd92"}
Sep 15 18:51:03.078697 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:03.077984    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-28.ec2.internal" event={"ID":"f8d47b24220b0dba80a2c6cc3e11210a","Type":"ContainerStarted","Data":"8494ced915d7913c11cae75980d3e55b13877277492adc4a0df7c97b4184967b"}
Sep 15 18:51:03.092221 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:03.091896    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-28.ec2.internal" podStartSLOduration=3.091876414 podStartE2EDuration="3.091876414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:03.091312486 +0000 UTC m=+4.578641476" watchObservedRunningTime="2026-09-15 18:51:03.091876414 +0000 UTC m=+4.579205406"
Sep 15 18:51:03.097505 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:03.097342    2555 generic.go:356] "Generic (PLEG): container finished" podID="44b7a7d82c3ab405a9df12dc028af9e9" containerID="f8a94d589042737a8eecb27acf3d7878763723f6c2b77bb2aea607543b01fb94" exitCode=0
Sep 15 18:51:03.097505 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:03.097443    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal" event={"ID":"44b7a7d82c3ab405a9df12dc028af9e9","Type":"ContainerDied","Data":"f8a94d589042737a8eecb27acf3d7878763723f6c2b77bb2aea607543b01fb94"}
Sep 15 18:51:03.128352 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:03.128268    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xfzjp" event={"ID":"5cf60c84-6ebf-48cc-b094-f865d877bc03","Type":"ContainerStarted","Data":"9f690613ffcfd181e02db5435fd4708433efd5ea7c7e5eb524b779c8ecf90f2d"}
Sep 15 18:51:03.134255 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:03.134170    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l" event={"ID":"2077960a-71c3-4fb4-8220-9fc0bf5f40e0","Type":"ContainerStarted","Data":"897ff0f321a7d2b4a528659e9595dc077c1695051382b4c159bee8ff4ec7fe81"}
Sep 15 18:51:03.139090 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:03.139022    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bf8kf" event={"ID":"5de8260e-f158-4eca-a056-8e5e039c45cc","Type":"ContainerStarted","Data":"c2e8a38937d65ddf12a755fc1bb90ced27a5a6b9f6939201cb0b203b928336ed"}
Sep 15 18:51:04.160527 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:04.160443    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal" event={"ID":"44b7a7d82c3ab405a9df12dc028af9e9","Type":"ContainerStarted","Data":"d0208a7f4aaf9d6ad4ab49aea2af8697ca338543413a6fdb80da8e5eb390c791"}
Sep 15 18:51:04.177124 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:04.177069    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-28.ec2.internal" podStartSLOduration=4.1770487 podStartE2EDuration="4.1770487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:04.175924227 +0000 UTC m=+5.663253217" watchObservedRunningTime="2026-09-15 18:51:04.1770487 +0000 UTC m=+5.664377691"
Sep 15 18:51:04.563244 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:04.563149    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret\") pod \"global-pull-secret-syncer-j8kt6\" (UID: \"f16c1599-2406-48bc-88fb-36e9670dc851\") " pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:04.563418 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:04.563346    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:04.563418 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:04.563412    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret podName:f16c1599-2406-48bc-88fb-36e9670dc851 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:08.563393477 +0000 UTC m=+10.050722441 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret") pod "global-pull-secret-syncer-j8kt6" (UID: "f16c1599-2406-48bc-88fb-36e9670dc851") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:04.664835 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:04.664277    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8j6\" (UniqueName: \"kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6\") pod \"network-check-target-wxhxs\" (UID: \"f9e0b82d-6132-4018-9b73-cb4bb6639ba2\") " pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:04.664835 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:04.664373    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs\") pod \"network-metrics-daemon-lpczh\" (UID: \"0a36af7f-abf7-414c-80b2-6314083e1a60\") " pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:04.664835 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:04.664512    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:04.664835 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:04.664576    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs podName:0a36af7f-abf7-414c-80b2-6314083e1a60 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:08.664556288 +0000 UTC m=+10.151885271 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs") pod "network-metrics-daemon-lpczh" (UID: "0a36af7f-abf7-414c-80b2-6314083e1a60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:04.665277 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:04.664994    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:51:04.665277 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:04.665020    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:51:04.665277 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:04.665034    2555 projected.go:196] Error preparing data for projected volume kube-api-access-2c8j6 for pod openshift-network-diagnostics/network-check-target-wxhxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:04.665277 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:04.665086    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6 podName:f9e0b82d-6132-4018-9b73-cb4bb6639ba2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:08.665068719 +0000 UTC m=+10.152397686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2c8j6" (UniqueName: "kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6") pod "network-check-target-wxhxs" (UID: "f9e0b82d-6132-4018-9b73-cb4bb6639ba2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:05.034874 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:05.034790    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:05.034874 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:05.034818    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:05.035136 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:05.034801    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:05.035136 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:05.034939    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:05.035136 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:05.035039    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:05.035298 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:05.035148    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:07.034487 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:07.034450    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:07.034987 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:07.034578    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:07.034987 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:07.034918    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:07.035103 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:07.035030    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:07.035103 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:07.035087    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:07.035216 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:07.035172    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:08.595964 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:08.595912    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret\") pod \"global-pull-secret-syncer-j8kt6\" (UID: \"f16c1599-2406-48bc-88fb-36e9670dc851\") " pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:08.596417 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:08.596046    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:08.596417 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:08.596118    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret podName:f16c1599-2406-48bc-88fb-36e9670dc851 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:16.596097178 +0000 UTC m=+18.083426153 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret") pod "global-pull-secret-syncer-j8kt6" (UID: "f16c1599-2406-48bc-88fb-36e9670dc851") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:08.696817 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:08.696781    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs\") pod \"network-metrics-daemon-lpczh\" (UID: \"0a36af7f-abf7-414c-80b2-6314083e1a60\") " pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:08.696984 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:08.696851    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8j6\" (UniqueName: \"kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6\") pod \"network-check-target-wxhxs\" (UID: \"f9e0b82d-6132-4018-9b73-cb4bb6639ba2\") " pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:08.697067 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:08.697008    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:51:08.697067 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:08.697037    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:51:08.697067 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:08.697046    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:08.697220 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:08.697127    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs podName:0a36af7f-abf7-414c-80b2-6314083e1a60 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:16.697104151 +0000 UTC m=+18.184433118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs") pod "network-metrics-daemon-lpczh" (UID: "0a36af7f-abf7-414c-80b2-6314083e1a60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:08.697220 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:08.697052    2555 projected.go:196] Error preparing data for projected volume kube-api-access-2c8j6 for pod openshift-network-diagnostics/network-check-target-wxhxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:08.697341 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:08.697222    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6 podName:f9e0b82d-6132-4018-9b73-cb4bb6639ba2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:16.697202788 +0000 UTC m=+18.184531758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2c8j6" (UniqueName: "kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6") pod "network-check-target-wxhxs" (UID: "f9e0b82d-6132-4018-9b73-cb4bb6639ba2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:09.035987 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:09.035438    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:09.035987 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:09.035536    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:09.035987 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:09.035553    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:09.035987 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:09.035722    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:09.035987 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:09.035788    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:09.035987 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:09.035877    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:11.034039 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:11.033999    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:11.034458 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:11.034130    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:11.034458 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:11.034337    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:11.034583 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:11.034471    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:11.034583 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:11.034515    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:11.034698 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:11.034634    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:13.033863 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:13.033645    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:13.034293 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:13.033653    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:13.034293 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:13.033954    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:13.034293 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:13.033697    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:13.034293 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:13.034034    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:13.034293 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:13.034109    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:15.034351 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:15.034314    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:15.034846 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:15.034322    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:15.034846 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:15.034475    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:15.034846 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:15.034545    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:15.034846 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:15.034322    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:15.034846 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:15.034654    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:16.656821 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:16.656775    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret\") pod \"global-pull-secret-syncer-j8kt6\" (UID: \"f16c1599-2406-48bc-88fb-36e9670dc851\") " pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:16.657266 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:16.656955    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:16.657266 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:16.657042    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret podName:f16c1599-2406-48bc-88fb-36e9670dc851 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:32.657019086 +0000 UTC m=+34.144348064 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret") pod "global-pull-secret-syncer-j8kt6" (UID: "f16c1599-2406-48bc-88fb-36e9670dc851") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:16.757470 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:16.757430    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs\") pod \"network-metrics-daemon-lpczh\" (UID: \"0a36af7f-abf7-414c-80b2-6314083e1a60\") " pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:16.757640 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:16.757498    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8j6\" (UniqueName: \"kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6\") pod \"network-check-target-wxhxs\" (UID: \"f9e0b82d-6132-4018-9b73-cb4bb6639ba2\") " pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:16.757640 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:16.757618    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:16.757738 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:16.757660    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:51:16.757738 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:16.757680    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:51:16.757738 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:16.757692    2555 projected.go:196] Error preparing data for projected volume kube-api-access-2c8j6 for pod openshift-network-diagnostics/network-check-target-wxhxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:16.757738 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:16.757698    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs podName:0a36af7f-abf7-414c-80b2-6314083e1a60 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:32.757677439 +0000 UTC m=+34.245006420 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs") pod "network-metrics-daemon-lpczh" (UID: "0a36af7f-abf7-414c-80b2-6314083e1a60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:16.757884 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:16.757741    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6 podName:f9e0b82d-6132-4018-9b73-cb4bb6639ba2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:32.757725157 +0000 UTC m=+34.245054125 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2c8j6" (UniqueName: "kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6") pod "network-check-target-wxhxs" (UID: "f9e0b82d-6132-4018-9b73-cb4bb6639ba2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:17.034353 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:17.034325    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:17.034510 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:17.034368    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:17.034510 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:17.034326    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:17.034510 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:17.034446    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:17.034603 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:17.034504    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:17.034603 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:17.034574    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:19.034697 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:19.034656    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:19.035151 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:19.034782    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:19.035151 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:19.034792    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:19.035151 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:19.034814    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:19.035151 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:19.034926    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:19.035151 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:19.035016    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:21.034129 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.033886    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:21.034742 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.033886    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:21.034742 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:21.034234    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:21.034742 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:21.034354    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:21.034742 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.033930    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:21.034742 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:21.034449    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:21.203751 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.203723    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xfzjp" event={"ID":"5cf60c84-6ebf-48cc-b094-f865d877bc03","Type":"ContainerStarted","Data":"5041daab91154dbf7d9c4262fd2ea05838cfaebc91411f14cef4ba5475fed6c7"}
Sep 15 18:51:21.205177 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.205156    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l" event={"ID":"2077960a-71c3-4fb4-8220-9fc0bf5f40e0","Type":"ContainerStarted","Data":"4012ca8dcd6e481e5caa09d91e9e42f42c3972eea026a760924ba6fc0abd2676"}
Sep 15 18:51:21.210543 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.210519    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz" event={"ID":"f9777e14-6d89-422a-8c73-89744f2a4020","Type":"ContainerStarted","Data":"5ed281f6b3b18123dae6372241bb128a6fde58a241fe8eca172674d7b6c759fb"}
Sep 15 18:51:21.210712 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.210637    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz" event={"ID":"f9777e14-6d89-422a-8c73-89744f2a4020","Type":"ContainerStarted","Data":"2ca480bb4c0dc775671eeced919e8f0fc3a09688a7811c6d6a40d7a017a05876"}
Sep 15 18:51:21.210712 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.210690    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz" event={"ID":"f9777e14-6d89-422a-8c73-89744f2a4020","Type":"ContainerStarted","Data":"4c7c7e980d428527b14d2dc33788ddd7405179e050116bcc91ca840c6c73b8a5"}
Sep 15 18:51:21.210805 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.210722    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz" event={"ID":"f9777e14-6d89-422a-8c73-89744f2a4020","Type":"ContainerStarted","Data":"aee6d7c2c657b218bfa2117b4257102545db027bb6cf5713fb7f0e874a3b12a0"}
Sep 15 18:51:21.210805 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.210759    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz" event={"ID":"f9777e14-6d89-422a-8c73-89744f2a4020","Type":"ContainerStarted","Data":"dd58e15f2a05489e3b89b75dfd11854527c73e1f9f2b541fe92d4b312aac5bac"}
Sep 15 18:51:21.212070 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.212046    2555 generic.go:356] "Generic (PLEG): container finished" podID="598c3d78-c05f-42dc-ac4f-0a6640c6fc38" containerID="0760685a678b138f1e7465c0c46a52a3feffd064ab94e1b0d299d0aeff215882" exitCode=0
Sep 15 18:51:21.212158 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.212135    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sq4v" event={"ID":"598c3d78-c05f-42dc-ac4f-0a6640c6fc38","Type":"ContainerDied","Data":"0760685a678b138f1e7465c0c46a52a3feffd064ab94e1b0d299d0aeff215882"}
Sep 15 18:51:21.213632 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.213596    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g2fwg" event={"ID":"4824dade-a75e-4ddc-b086-989eaedafe12","Type":"ContainerStarted","Data":"7e6bbace4f90bf6966cfc03e3f8408e5a9001ae08b529fdd168bc9bb85b03b26"}
Sep 15 18:51:21.215140 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.215115    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d8r6c" event={"ID":"c9555dd5-d766-4b43-bd0b-e2ed73acc9cd","Type":"ContainerStarted","Data":"1fbc07359d01aeb0d5d27a0ffbc5e5e9ae7c84aa9dcda994fe13b9b307a75164"}
Sep 15 18:51:21.216730 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.216709    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b2mzg" event={"ID":"fc83f4f3-e01c-4e16-86cd-8eadc1ad6307","Type":"ContainerStarted","Data":"5b6415a847b972ca03e854956a883d3512b6cf6e0ab22964e6f1d6c071042afd"}
Sep 15 18:51:21.218189 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.218165    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tdwks" event={"ID":"cd565903-2276-469c-a07c-63cd015ef00a","Type":"ContainerStarted","Data":"492d5dfb9423aeb35c1c917350384f7eeb12e08c1e97a6153cf5a51aa8e9f524"}
Sep 15 18:51:21.223309 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.223269    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xfzjp" podStartSLOduration=3.886441227 podStartE2EDuration="22.223257848s" totalImagesPullingTime="18.336816621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:21.222638813 +0000 UTC m=+22.709967799" watchObservedRunningTime="2026-09-15 18:51:21.223257848 +0000 UTC m=+22.710586850"
Sep 15 18:51:21.236126 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.236087    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g2fwg" podStartSLOduration=2.947758526 podStartE2EDuration="21.23607514s" totalImagesPullingTime="18.288316614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:21.235582869 +0000 UTC m=+22.722911853" watchObservedRunningTime="2026-09-15 18:51:21.23607514 +0000 UTC m=+22.723404125"
Sep 15 18:51:21.270734 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.270689    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tdwks" podStartSLOduration=6.616778491 podStartE2EDuration="21.270674203s" totalImagesPullingTime="14.653895712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:21.270228097 +0000 UTC m=+22.757557082" watchObservedRunningTime="2026-09-15 18:51:21.270674203 +0000 UTC m=+22.758003188"
Sep 15 18:51:21.284635 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.284567    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-d8r6c" podStartSLOduration=6.641054571 podStartE2EDuration="21.284549924s" totalImagesPullingTime="14.643495353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:21.284495157 +0000 UTC m=+22.771824143" watchObservedRunningTime="2026-09-15 18:51:21.284549924 +0000 UTC m=+22.771878910"
Sep 15 18:51:21.302913 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:21.302872    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b2mzg" podStartSLOduration=3.004826604 podStartE2EDuration="21.302858445s" totalImagesPullingTime="18.298031841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:21.302760209 +0000 UTC m=+22.790089195" watchObservedRunningTime="2026-09-15 18:51:21.302858445 +0000 UTC m=+22.790187430"
Sep 15 18:51:22.222206 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:22.222126    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bf8kf" event={"ID":"5de8260e-f158-4eca-a056-8e5e039c45cc","Type":"ContainerStarted","Data":"39c2a8f437a3cba377d3b79b66a70317f5ceb814d1ee17a863da4a73f2439cbf"}
Sep 15 18:51:22.228600 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:22.228572    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz" event={"ID":"f9777e14-6d89-422a-8c73-89744f2a4020","Type":"ContainerStarted","Data":"de31aa63c3de0ddf03e597f00d1535b125a40629adaa65b8e1cb4ff04bc98904"}
Sep 15 18:51:22.236061 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:22.236016    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bf8kf" podStartSLOduration=3.9452081039999998 podStartE2EDuration="22.235998771s" totalImagesPullingTime="18.290790667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:22.235311282 +0000 UTC m=+23.722640269" watchObservedRunningTime="2026-09-15 18:51:22.235998771 +0000 UTC m=+23.723327756"
Sep 15 18:51:22.304804 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:22.304583    2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 18:51:22.970340 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:22.970238    2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T18:51:22.30479964Z","UUID":"5a5686f4-b5dd-448f-a2a1-ce25153e8a73","Handler":null,"Name":"","Endpoint":""}
Sep 15 18:51:22.972391 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:22.972370    2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 18:51:22.972544 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:22.972399    2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 18:51:23.033911 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:23.033877    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:23.034068 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:23.033996    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:23.034130 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:23.034093    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:23.034272 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:23.034251    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:23.034376 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:23.034268    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:23.034457 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:23.034430    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:23.233476 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:23.233391    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l" event={"ID":"2077960a-71c3-4fb4-8220-9fc0bf5f40e0","Type":"ContainerStarted","Data":"97d2dd50e1bcd9509d48af8cf3aae3b017e4220bdf3cd81adfa87a66a75c7fb7"}
Sep 15 18:51:24.235811 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:24.235741    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tdwks"
Sep 15 18:51:24.236439 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:24.236419    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tdwks"
Sep 15 18:51:24.238671 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:24.238641    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l" event={"ID":"2077960a-71c3-4fb4-8220-9fc0bf5f40e0","Type":"ContainerStarted","Data":"3ac0e0ec0e876e5f79fc3404ae71161a9472fd9228c4998f055ade6030941e20"}
Sep 15 18:51:24.242564 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:24.242542    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz" event={"ID":"f9777e14-6d89-422a-8c73-89744f2a4020","Type":"ContainerStarted","Data":"a377834c66d20830f7d80811cba5b726fe568568782b092c26a0888b57d147fc"}
Sep 15 18:51:25.034081 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:25.034047    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:25.034081 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:25.034077    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:25.034327 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:25.034051    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:25.034327 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:25.034177    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:25.034327 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:25.034263    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:25.034490 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:25.034401    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:25.244868 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:25.244836    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tdwks"
Sep 15 18:51:25.245337 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:25.245225    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tdwks"
Sep 15 18:51:25.258026 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:25.257979    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkj5l" podStartSLOduration=5.619693208 podStartE2EDuration="26.257963888s" totalImagesPullingTime="20.63827068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:24.281304988 +0000 UTC m=+25.768633973" watchObservedRunningTime="2026-09-15 18:51:25.257963888 +0000 UTC m=+26.745292889"
Sep 15 18:51:26.250560 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:26.250523    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz" event={"ID":"f9777e14-6d89-422a-8c73-89744f2a4020","Type":"ContainerStarted","Data":"3d9e963e4f171357aed7e708894f4c9d0706aa09546c3f724e54a252fbd0bdf8"}
Sep 15 18:51:26.251139 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:26.250800    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:26.251139 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:26.250854    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:26.252505 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:26.252483    2555 generic.go:356] "Generic (PLEG): container finished" podID="598c3d78-c05f-42dc-ac4f-0a6640c6fc38" containerID="2c7a77161de3807611afc7374e1c2387761b883b483312efb47e9e6941f43c7c" exitCode=0
Sep 15 18:51:26.252664 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:26.252553    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sq4v" event={"ID":"598c3d78-c05f-42dc-ac4f-0a6640c6fc38","Type":"ContainerDied","Data":"2c7a77161de3807611afc7374e1c2387761b883b483312efb47e9e6941f43c7c"}
Sep 15 18:51:26.265907 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:26.265886    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:26.274006 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:26.273966    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz" podStartSLOduration=8.926050734 podStartE2EDuration="27.273952389s" totalImagesPullingTime="18.347901655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:26.273529891 +0000 UTC m=+27.760858875" watchObservedRunningTime="2026-09-15 18:51:26.273952389 +0000 UTC m=+27.761281374"
Sep 15 18:51:27.037355 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:27.037323    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:27.037536 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:27.037414    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:27.037536 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:27.037420    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:27.037536 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:27.037447    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:27.037701 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:27.037535    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:27.037701 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:27.037636    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:27.255210 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:27.255181    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:27.269278 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:27.269254    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:28.258686 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:28.258649    2555 generic.go:356] "Generic (PLEG): container finished" podID="598c3d78-c05f-42dc-ac4f-0a6640c6fc38" containerID="19defbd115f7bc0a35994dafef2d21ea6ef989128177e662e88697c9f128a39d" exitCode=0
Sep 15 18:51:28.259169 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:28.258697    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sq4v" event={"ID":"598c3d78-c05f-42dc-ac4f-0a6640c6fc38","Type":"ContainerDied","Data":"19defbd115f7bc0a35994dafef2d21ea6ef989128177e662e88697c9f128a39d"}
Sep 15 18:51:29.034772 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:29.034744    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:29.034926 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:29.034825    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:29.034926 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:29.034855    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:29.034994 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:29.034941    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:29.034994 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:29.034966    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:29.035052 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:29.035042    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:30.265202 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:30.265167    2555 generic.go:356] "Generic (PLEG): container finished" podID="598c3d78-c05f-42dc-ac4f-0a6640c6fc38" containerID="2e67e59ae18aee60f1765b5864eaf945576d4a505d578ec496250db030f8e59b" exitCode=0
Sep 15 18:51:30.265691 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:30.265218    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sq4v" event={"ID":"598c3d78-c05f-42dc-ac4f-0a6640c6fc38","Type":"ContainerDied","Data":"2e67e59ae18aee60f1765b5864eaf945576d4a505d578ec496250db030f8e59b"}
Sep 15 18:51:31.033511 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:31.033475    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:31.033788 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:31.033579    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:31.033788 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:31.033586    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:31.033788 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:31.033726    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:31.033788 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:31.033760    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:31.033970 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:31.033832    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:32.677283 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:32.677240    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret\") pod \"global-pull-secret-syncer-j8kt6\" (UID: \"f16c1599-2406-48bc-88fb-36e9670dc851\") " pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:32.677800 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:32.677365    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:32.677800 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:32.677424    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret podName:f16c1599-2406-48bc-88fb-36e9670dc851 nodeName:}" failed. No retries permitted until 2026-09-15 18:52:04.677411176 +0000 UTC m=+66.164740139 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret") pod "global-pull-secret-syncer-j8kt6" (UID: "f16c1599-2406-48bc-88fb-36e9670dc851") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:32.777542 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:32.777511    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs\") pod \"network-metrics-daemon-lpczh\" (UID: \"0a36af7f-abf7-414c-80b2-6314083e1a60\") " pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:32.777742 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:32.777568    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8j6\" (UniqueName: \"kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6\") pod \"network-check-target-wxhxs\" (UID: \"f9e0b82d-6132-4018-9b73-cb4bb6639ba2\") " pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:32.777742 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:32.777647    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:32.777742 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:32.777694    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:51:32.777742 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:32.777711    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:51:32.777742 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:32.777717    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs podName:0a36af7f-abf7-414c-80b2-6314083e1a60 nodeName:}" failed. No retries permitted until 2026-09-15 18:52:04.777699853 +0000 UTC m=+66.265028829 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs") pod "network-metrics-daemon-lpczh" (UID: "0a36af7f-abf7-414c-80b2-6314083e1a60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:32.777742 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:32.777724    2555 projected.go:196] Error preparing data for projected volume kube-api-access-2c8j6 for pod openshift-network-diagnostics/network-check-target-wxhxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:32.778066 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:32.777759    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6 podName:f9e0b82d-6132-4018-9b73-cb4bb6639ba2 nodeName:}" failed. No retries permitted until 2026-09-15 18:52:04.777750044 +0000 UTC m=+66.265079022 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2c8j6" (UniqueName: "kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6") pod "network-check-target-wxhxs" (UID: "f9e0b82d-6132-4018-9b73-cb4bb6639ba2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:33.034000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:33.033968    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:33.034184 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:33.033968    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:33.034184 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:33.034110    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:33.034184 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:33.033968    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:33.034537 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:33.034348    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:33.034640 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:33.034627    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:35.033582 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:35.033551    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:35.033582 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:35.033571    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:35.034060 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:35.033560    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:35.034060 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:35.033665    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:35.034060 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:35.033753    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:35.034060 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:35.033848    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:35.281090 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:35.281061    2555 generic.go:356] "Generic (PLEG): container finished" podID="598c3d78-c05f-42dc-ac4f-0a6640c6fc38" containerID="47fbbc8092d850e3583a0a486b9f3f237b55abbd3a948bc931300b253ce34cc2" exitCode=0
Sep 15 18:51:35.281255 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:35.281103    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sq4v" event={"ID":"598c3d78-c05f-42dc-ac4f-0a6640c6fc38","Type":"ContainerDied","Data":"47fbbc8092d850e3583a0a486b9f3f237b55abbd3a948bc931300b253ce34cc2"}
Sep 15 18:51:36.286735 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:36.286531    2555 generic.go:356] "Generic (PLEG): container finished" podID="598c3d78-c05f-42dc-ac4f-0a6640c6fc38" containerID="e4cf7f8c95186d533c09747d7e1d54a533360b764c70a253147c94a006a9004d" exitCode=0
Sep 15 18:51:36.286735 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:36.286624    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sq4v" event={"ID":"598c3d78-c05f-42dc-ac4f-0a6640c6fc38","Type":"ContainerDied","Data":"e4cf7f8c95186d533c09747d7e1d54a533360b764c70a253147c94a006a9004d"}
Sep 15 18:51:37.034558 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:37.034517    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:37.034746 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:37.034576    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:37.034746 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:37.034579    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:37.034746 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:37.034672    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:37.034933 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:37.034755    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:37.034933 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:37.034823    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:37.292419 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:37.292329    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sq4v" event={"ID":"598c3d78-c05f-42dc-ac4f-0a6640c6fc38","Type":"ContainerStarted","Data":"6df5ad2c488deaa6c03638d4dd6ea9da9933ed082547113a1a2bb80a84a859e2"}
Sep 15 18:51:37.313253 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:37.313200    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2sq4v" podStartSLOduration=9.136744391 podStartE2EDuration="38.313185458s" totalImagesPullingTime="29.176441067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:37.311881498 +0000 UTC m=+38.799210483" watchObservedRunningTime="2026-09-15 18:51:37.313185458 +0000 UTC m=+38.800514443"
Sep 15 18:51:39.037688 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:39.036459    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:39.037688 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:39.036730    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:39.037688 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:39.037041    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:39.037688 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:39.037196    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:39.037688 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:39.037386    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:39.037688 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:39.037491    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:40.478496 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:40.478462    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j8kt6"]
Sep 15 18:51:40.478937 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:40.478584    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:40.478937 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:40.478712    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:40.481525 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:40.481483    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wxhxs"]
Sep 15 18:51:40.481812 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:40.481793    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:40.482049 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:40.482019    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:40.482252 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:40.482228    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lpczh"]
Sep 15 18:51:40.482483 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:40.482467    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:40.482708 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:40.482688    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:42.033653 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:42.033605    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:42.034101 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:42.033623    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:42.034101 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:42.033727    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:42.034101 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:42.033828    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:42.034101 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:42.033621    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:42.034101 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:42.033923    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:44.033492 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.033461    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:44.033873 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.033468    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:44.033873 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:44.033564    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8kt6" podUID="f16c1599-2406-48bc-88fb-36e9670dc851"
Sep 15 18:51:44.033873 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.033576    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:44.033873 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:44.033679    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxhxs" podUID="f9e0b82d-6132-4018-9b73-cb4bb6639ba2"
Sep 15 18:51:44.033873 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:44.033744    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lpczh" podUID="0a36af7f-abf7-414c-80b2-6314083e1a60"
Sep 15 18:51:44.247317 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.247292    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-28.ec2.internal" event="NodeReady"
Sep 15 18:51:44.247510 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.247440    2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 18:51:44.292025 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.291951    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rldqf"]
Sep 15 18:51:44.309032 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.309010    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zknv8"]
Sep 15 18:51:44.309219 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.309160    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rldqf"
Sep 15 18:51:44.311356 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.311334    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 18:51:44.311486 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.311365    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 18:51:44.311572 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.311530    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vcddk\""
Sep 15 18:51:44.322999 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.322926    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rldqf"]
Sep 15 18:51:44.323110 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.323010    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zknv8"]
Sep 15 18:51:44.323110 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.323062    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zknv8"
Sep 15 18:51:44.325058 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.325040    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 18:51:44.325184 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.325127    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 18:51:44.332140 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.332118    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 18:51:44.332140 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.332130    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6fg9f\""
Sep 15 18:51:44.395461 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.395430    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wh7k8"]
Sep 15 18:51:44.416382 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.416357    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wh7k8"]
Sep 15 18:51:44.416514 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.416474    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wh7k8"
Sep 15 18:51:44.418879 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.418849    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 18:51:44.418879 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.418874    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 18:51:44.419069 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.418930    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 18:51:44.419069 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.418982    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 18:51:44.419225 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.419203    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pj5l2\""
Sep 15 18:51:44.453713 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.453691    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48b28e4d-19dd-43ea-ae96-cd04ac848020-tmp-dir\") pod \"dns-default-rldqf\" (UID: \"48b28e4d-19dd-43ea-ae96-cd04ac848020\") " pod="openshift-dns/dns-default-rldqf"
Sep 15 18:51:44.453843 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.453726    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48b28e4d-19dd-43ea-ae96-cd04ac848020-metrics-tls\") pod \"dns-default-rldqf\" (UID: \"48b28e4d-19dd-43ea-ae96-cd04ac848020\") " pod="openshift-dns/dns-default-rldqf"
Sep 15 18:51:44.453843 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.453743    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87nd5\" (UniqueName: \"kubernetes.io/projected/48b28e4d-19dd-43ea-ae96-cd04ac848020-kube-api-access-87nd5\") pod \"dns-default-rldqf\" (UID: \"48b28e4d-19dd-43ea-ae96-cd04ac848020\") " pod="openshift-dns/dns-default-rldqf"
Sep 15 18:51:44.453843 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.453768    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/48b28e4d-19dd-43ea-ae96-cd04ac848020-config-volume\") pod \"dns-default-rldqf\" (UID: \"48b28e4d-19dd-43ea-ae96-cd04ac848020\") " pod="openshift-dns/dns-default-rldqf"
Sep 15 18:51:44.453843 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.453810    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c998a93-b454-41f5-a589-87f5364c062c-cert\") pod \"ingress-canary-zknv8\" (UID: \"9c998a93-b454-41f5-a589-87f5364c062c\") " pod="openshift-ingress-canary/ingress-canary-zknv8"
Sep 15 18:51:44.453969 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.453868    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7tvp\" (UniqueName: \"kubernetes.io/projected/9c998a93-b454-41f5-a589-87f5364c062c-kube-api-access-z7tvp\") pod \"ingress-canary-zknv8\" (UID: \"9c998a93-b454-41f5-a589-87f5364c062c\") " pod="openshift-ingress-canary/ingress-canary-zknv8"
Sep 15 18:51:44.555058 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.554989    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7tvp\" (UniqueName: \"kubernetes.io/projected/9c998a93-b454-41f5-a589-87f5364c062c-kube-api-access-z7tvp\") pod \"ingress-canary-zknv8\" (UID: \"9c998a93-b454-41f5-a589-87f5364c062c\") " pod="openshift-ingress-canary/ingress-canary-zknv8"
Sep 15 18:51:44.555058 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.555032    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/afa9ab3a-3301-4822-b409-2cbbdb957b28-crio-socket\") pod \"insights-runtime-extractor-wh7k8\" (UID: \"afa9ab3a-3301-4822-b409-2cbbdb957b28\") " pod="openshift-insights/insights-runtime-extractor-wh7k8"
Sep 15 18:51:44.555184 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.555056    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r2hm\" (UniqueName: \"kubernetes.io/projected/afa9ab3a-3301-4822-b409-2cbbdb957b28-kube-api-access-2r2hm\") pod \"insights-runtime-extractor-wh7k8\" (UID: \"afa9ab3a-3301-4822-b409-2cbbdb957b28\") " pod="openshift-insights/insights-runtime-extractor-wh7k8"
Sep 15 18:51:44.555184 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.555111    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48b28e4d-19dd-43ea-ae96-cd04ac848020-tmp-dir\") pod \"dns-default-rldqf\" (UID: \"48b28e4d-19dd-43ea-ae96-cd04ac848020\") " pod="openshift-dns/dns-default-rldqf"
Sep 15 18:51:44.555184 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.555137    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48b28e4d-19dd-43ea-ae96-cd04ac848020-metrics-tls\") pod \"dns-default-rldqf\" (UID: \"48b28e4d-19dd-43ea-ae96-cd04ac848020\") " pod="openshift-dns/dns-default-rldqf"
Sep 15 18:51:44.555184 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.555155    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87nd5\" (UniqueName: \"kubernetes.io/projected/48b28e4d-19dd-43ea-ae96-cd04ac848020-kube-api-access-87nd5\") pod \"dns-default-rldqf\" (UID: \"48b28e4d-19dd-43ea-ae96-cd04ac848020\") " pod="openshift-dns/dns-default-rldqf"
Sep 15 18:51:44.555343 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.555195    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/afa9ab3a-3301-4822-b409-2cbbdb957b28-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wh7k8\" (UID: \"afa9ab3a-3301-4822-b409-2cbbdb957b28\") " pod="openshift-insights/insights-runtime-extractor-wh7k8"
Sep 15 18:51:44.555343 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.555221    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/48b28e4d-19dd-43ea-ae96-cd04ac848020-config-volume\") pod \"dns-default-rldqf\" (UID: \"48b28e4d-19dd-43ea-ae96-cd04ac848020\") " pod="openshift-dns/dns-default-rldqf"
Sep 15 18:51:44.555343 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.555243    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c998a93-b454-41f5-a589-87f5364c062c-cert\") pod \"ingress-canary-zknv8\" (UID: \"9c998a93-b454-41f5-a589-87f5364c062c\") " pod="openshift-ingress-canary/ingress-canary-zknv8"
Sep 15 18:51:44.555343 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.555265    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/afa9ab3a-3301-4822-b409-2cbbdb957b28-data-volume\") pod \"insights-runtime-extractor-wh7k8\" (UID: \"afa9ab3a-3301-4822-b409-2cbbdb957b28\") " pod="openshift-insights/insights-runtime-extractor-wh7k8"
Sep 15 18:51:44.555343 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.555287    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afa9ab3a-3301-4822-b409-2cbbdb957b28-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wh7k8\" (UID: \"afa9ab3a-3301-4822-b409-2cbbdb957b28\") " pod="openshift-insights/insights-runtime-extractor-wh7k8"
Sep 15 18:51:44.555883 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.555743    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48b28e4d-19dd-43ea-ae96-cd04ac848020-tmp-dir\") pod \"dns-default-rldqf\" (UID: \"48b28e4d-19dd-43ea-ae96-cd04ac848020\") " pod="openshift-dns/dns-default-rldqf"
Sep 15 18:51:44.556000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.555977    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/48b28e4d-19dd-43ea-ae96-cd04ac848020-config-volume\") pod \"dns-default-rldqf\" (UID: \"48b28e4d-19dd-43ea-ae96-cd04ac848020\") " pod="openshift-dns/dns-default-rldqf"
Sep 15 18:51:44.559166 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.559139    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c998a93-b454-41f5-a589-87f5364c062c-cert\") pod \"ingress-canary-zknv8\" (UID: \"9c998a93-b454-41f5-a589-87f5364c062c\") " pod="openshift-ingress-canary/ingress-canary-zknv8"
Sep 15 18:51:44.559166 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.559157    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48b28e4d-19dd-43ea-ae96-cd04ac848020-metrics-tls\") pod \"dns-default-rldqf\" (UID: \"48b28e4d-19dd-43ea-ae96-cd04ac848020\") " pod="openshift-dns/dns-default-rldqf"
Sep 15 18:51:44.562391 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.562370    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7tvp\" (UniqueName: \"kubernetes.io/projected/9c998a93-b454-41f5-a589-87f5364c062c-kube-api-access-z7tvp\") pod \"ingress-canary-zknv8\" (UID: \"9c998a93-b454-41f5-a589-87f5364c062c\") " pod="openshift-ingress-canary/ingress-canary-zknv8"
Sep 15 18:51:44.562535 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.562514    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87nd5\" (UniqueName: \"kubernetes.io/projected/48b28e4d-19dd-43ea-ae96-cd04ac848020-kube-api-access-87nd5\") pod \"dns-default-rldqf\" (UID: \"48b28e4d-19dd-43ea-ae96-cd04ac848020\") " pod="openshift-dns/dns-default-rldqf"
Sep 15 18:51:44.619062 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.619024    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rldqf"
Sep 15 18:51:44.631168 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.631138    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zknv8"
Sep 15 18:51:44.655523 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.655497    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/afa9ab3a-3301-4822-b409-2cbbdb957b28-data-volume\") pod \"insights-runtime-extractor-wh7k8\" (UID: \"afa9ab3a-3301-4822-b409-2cbbdb957b28\") " pod="openshift-insights/insights-runtime-extractor-wh7k8"
Sep 15 18:51:44.655643 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.655527    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afa9ab3a-3301-4822-b409-2cbbdb957b28-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wh7k8\" (UID: \"afa9ab3a-3301-4822-b409-2cbbdb957b28\") " pod="openshift-insights/insights-runtime-extractor-wh7k8"
Sep 15 18:51:44.655643 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.655555    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/afa9ab3a-3301-4822-b409-2cbbdb957b28-crio-socket\") pod \"insights-runtime-extractor-wh7k8\" (UID: \"afa9ab3a-3301-4822-b409-2cbbdb957b28\") " pod="openshift-insights/insights-runtime-extractor-wh7k8"
Sep 15 18:51:44.655643 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.655581    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r2hm\" (UniqueName: \"kubernetes.io/projected/afa9ab3a-3301-4822-b409-2cbbdb957b28-kube-api-access-2r2hm\") pod \"insights-runtime-extractor-wh7k8\" (UID: \"afa9ab3a-3301-4822-b409-2cbbdb957b28\") " pod="openshift-insights/insights-runtime-extractor-wh7k8"
Sep 15 18:51:44.655750 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.655732    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/afa9ab3a-3301-4822-b409-2cbbdb957b28-crio-socket\") pod \"insights-runtime-extractor-wh7k8\" (UID: \"afa9ab3a-3301-4822-b409-2cbbdb957b28\") " pod="openshift-insights/insights-runtime-extractor-wh7k8"
Sep 15 18:51:44.655790 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.655765    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/afa9ab3a-3301-4822-b409-2cbbdb957b28-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wh7k8\" (UID: \"afa9ab3a-3301-4822-b409-2cbbdb957b28\") " pod="openshift-insights/insights-runtime-extractor-wh7k8"
Sep 15 18:51:44.655888 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.655871    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/afa9ab3a-3301-4822-b409-2cbbdb957b28-data-volume\") pod \"insights-runtime-extractor-wh7k8\" (UID: \"afa9ab3a-3301-4822-b409-2cbbdb957b28\") " pod="openshift-insights/insights-runtime-extractor-wh7k8"
Sep 15 18:51:44.656228 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.656212    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/afa9ab3a-3301-4822-b409-2cbbdb957b28-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wh7k8\" (UID: \"afa9ab3a-3301-4822-b409-2cbbdb957b28\") " pod="openshift-insights/insights-runtime-extractor-wh7k8"
Sep 15 18:51:44.657916 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.657885    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/afa9ab3a-3301-4822-b409-2cbbdb957b28-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wh7k8\" (UID: \"afa9ab3a-3301-4822-b409-2cbbdb957b28\") " pod="openshift-insights/insights-runtime-extractor-wh7k8"
Sep 15 18:51:44.667340 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.666675    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r2hm\" (UniqueName: \"kubernetes.io/projected/afa9ab3a-3301-4822-b409-2cbbdb957b28-kube-api-access-2r2hm\") pod \"insights-runtime-extractor-wh7k8\" (UID: \"afa9ab3a-3301-4822-b409-2cbbdb957b28\") " pod="openshift-insights/insights-runtime-extractor-wh7k8"
Sep 15 18:51:44.726246 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.726212    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wh7k8"
Sep 15 18:51:44.959840 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.959809    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zknv8"]
Sep 15 18:51:44.961950 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:44.961922    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rldqf"]
Sep 15 18:51:44.962959 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:44.962938    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c998a93_b454_41f5_a589_87f5364c062c.slice/crio-081c6e958316c0e4c70d96657757da812c9c185124d61c57c285ba31261d878c WatchSource:0}: Error finding container 081c6e958316c0e4c70d96657757da812c9c185124d61c57c285ba31261d878c: Status 404 returned error can't find the container with id 081c6e958316c0e4c70d96657757da812c9c185124d61c57c285ba31261d878c
Sep 15 18:51:44.964789 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:44.964714    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b28e4d_19dd_43ea_ae96_cd04ac848020.slice/crio-c1fb04a22f071a963de46d24f9881ed4609fe4b070d9010db6039bf992a6fd08 WatchSource:0}: Error finding container c1fb04a22f071a963de46d24f9881ed4609fe4b070d9010db6039bf992a6fd08: Status 404 returned error can't find the container with id c1fb04a22f071a963de46d24f9881ed4609fe4b070d9010db6039bf992a6fd08
Sep 15 18:51:45.027561 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:45.027511    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wh7k8"]
Sep 15 18:51:45.029797 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:45.029776    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa9ab3a_3301_4822_b409_2cbbdb957b28.slice/crio-cfd0552500e1f0f0fe483c4c1732e717f4587975b3634f21ed74f4267aa72c91 WatchSource:0}: Error finding container cfd0552500e1f0f0fe483c4c1732e717f4587975b3634f21ed74f4267aa72c91: Status 404 returned error can't find the container with id cfd0552500e1f0f0fe483c4c1732e717f4587975b3634f21ed74f4267aa72c91
Sep 15 18:51:45.312842 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:45.312814    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wh7k8" event={"ID":"afa9ab3a-3301-4822-b409-2cbbdb957b28","Type":"ContainerStarted","Data":"35e7437375043e8f9a35864dc22fc76c9bba15be13032f0fe2451d78c75a64e4"}
Sep 15 18:51:45.313678 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:45.312870    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wh7k8" event={"ID":"afa9ab3a-3301-4822-b409-2cbbdb957b28","Type":"ContainerStarted","Data":"cfd0552500e1f0f0fe483c4c1732e717f4587975b3634f21ed74f4267aa72c91"}
Sep 15 18:51:45.314052 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:45.314032    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rldqf" event={"ID":"48b28e4d-19dd-43ea-ae96-cd04ac848020","Type":"ContainerStarted","Data":"c1fb04a22f071a963de46d24f9881ed4609fe4b070d9010db6039bf992a6fd08"}
Sep 15 18:51:45.315159 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:45.315136    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zknv8" event={"ID":"9c998a93-b454-41f5-a589-87f5364c062c","Type":"ContainerStarted","Data":"081c6e958316c0e4c70d96657757da812c9c185124d61c57c285ba31261d878c"}
Sep 15 18:51:46.034288 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:46.034254    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:51:46.034445 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:46.034260    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:51:46.034445 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:46.034260    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:51:46.037581 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:46.037552    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 18:51:46.037737 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:46.037552    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 18:51:46.037737 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:46.037630    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 18:51:46.037737 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:46.037664    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4h8bk\""
Sep 15 18:51:46.037737 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:46.037552    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 18:51:46.037737 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:46.037552    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gnvqj\""
Sep 15 18:51:47.934738 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:47.934483    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-x8s89"]
Sep 15 18:51:47.971107 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:47.971077    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-x8s89"]
Sep 15 18:51:47.971266 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:47.971194    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-x8s89"
Sep 15 18:51:47.974027 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:47.974005    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 18:51:47.974193 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:47.974117    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 15 18:51:47.974193 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:47.974150    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 18:51:47.975110 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:47.975092    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 15 18:51:47.975110 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:47.975102    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 18:51:47.975327 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:47.975151    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-xsncl\""
Sep 15 18:51:48.083889 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:48.083850    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9957ce75-f34c-42b9-804a-dd68f84af8d6-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-x8s89\" (UID: \"9957ce75-f34c-42b9-804a-dd68f84af8d6\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-x8s89"
Sep 15 18:51:48.084038 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:48.083940    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9957ce75-f34c-42b9-804a-dd68f84af8d6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-x8s89\" (UID: \"9957ce75-f34c-42b9-804a-dd68f84af8d6\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-x8s89"
Sep 15 18:51:48.084038 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:48.084021    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9957ce75-f34c-42b9-804a-dd68f84af8d6-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-x8s89\" (UID: \"9957ce75-f34c-42b9-804a-dd68f84af8d6\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-x8s89"
Sep 15 18:51:48.084153 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:48.084072    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clslk\" (UniqueName: \"kubernetes.io/projected/9957ce75-f34c-42b9-804a-dd68f84af8d6-kube-api-access-clslk\") pod \"prometheus-operator-68bcd7f987-x8s89\" (UID: \"9957ce75-f34c-42b9-804a-dd68f84af8d6\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-x8s89"
Sep 15 18:51:48.184819 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:48.184759    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9957ce75-f34c-42b9-804a-dd68f84af8d6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-x8s89\" (UID: \"9957ce75-f34c-42b9-804a-dd68f84af8d6\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-x8s89"
Sep 15 18:51:48.184819 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:48.184815    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9957ce75-f34c-42b9-804a-dd68f84af8d6-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-x8s89\" (UID: \"9957ce75-f34c-42b9-804a-dd68f84af8d6\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-x8s89"
Sep 15 18:51:48.184976 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:48.184845    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clslk\" (UniqueName: \"kubernetes.io/projected/9957ce75-f34c-42b9-804a-dd68f84af8d6-kube-api-access-clslk\") pod \"prometheus-operator-68bcd7f987-x8s89\" (UID: \"9957ce75-f34c-42b9-804a-dd68f84af8d6\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-x8s89"
Sep 15 18:51:48.184976 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:48.184870    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9957ce75-f34c-42b9-804a-dd68f84af8d6-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-x8s89\" (UID: \"9957ce75-f34c-42b9-804a-dd68f84af8d6\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-x8s89"
Sep 15 18:51:48.184976 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:48.184967    2555 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 15 18:51:48.185132 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:48.185022    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9957ce75-f34c-42b9-804a-dd68f84af8d6-prometheus-operator-tls podName:9957ce75-f34c-42b9-804a-dd68f84af8d6 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:48.685004944 +0000 UTC m=+50.172333912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/9957ce75-f34c-42b9-804a-dd68f84af8d6-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-x8s89" (UID: "9957ce75-f34c-42b9-804a-dd68f84af8d6") : secret "prometheus-operator-tls" not found
Sep 15 18:51:48.185631 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:48.185588    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9957ce75-f34c-42b9-804a-dd68f84af8d6-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-x8s89\" (UID: \"9957ce75-f34c-42b9-804a-dd68f84af8d6\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-x8s89"
Sep 15 18:51:48.188263 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:48.188232    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9957ce75-f34c-42b9-804a-dd68f84af8d6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-x8s89\" (UID: \"9957ce75-f34c-42b9-804a-dd68f84af8d6\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-x8s89"
Sep 15 18:51:48.192468 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:48.192446    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clslk\" (UniqueName: \"kubernetes.io/projected/9957ce75-f34c-42b9-804a-dd68f84af8d6-kube-api-access-clslk\") pod \"prometheus-operator-68bcd7f987-x8s89\" (UID: \"9957ce75-f34c-42b9-804a-dd68f84af8d6\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-x8s89"
Sep 15 18:51:48.324910 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:48.324878    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wh7k8" event={"ID":"afa9ab3a-3301-4822-b409-2cbbdb957b28","Type":"ContainerStarted","Data":"108214797e26fa6ca88345c1e0452b4a78db5d09804a3ff1a825e77156e1fb0c"}
Sep 15 18:51:48.326586 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:48.326560    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rldqf" event={"ID":"48b28e4d-19dd-43ea-ae96-cd04ac848020","Type":"ContainerStarted","Data":"70e6f8d54a86aebd3bf20690a2fbbc213a2efb3e4d2bc9f8f3ee2608a5f70c20"}
Sep 15 18:51:48.326705 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:48.326602    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rldqf" event={"ID":"48b28e4d-19dd-43ea-ae96-cd04ac848020","Type":"ContainerStarted","Data":"344014ac9aaf7654614162f87648fef776adadd7495b1faf752f99a570f67a87"}
Sep 15 18:51:48.326793 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:48.326730    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rldqf"
Sep 15 18:51:48.344017 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:48.343969    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rldqf" podStartSLOduration=1.276253936 podStartE2EDuration="4.343953772s" totalImagesPullingTime="3.067699836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:48.343235694 +0000 UTC m=+49.830564680" watchObservedRunningTime="2026-09-15 18:51:48.343953772 +0000 UTC m=+49.831282748"
Sep 15 18:51:48.688420 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:48.688382    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9957ce75-f34c-42b9-804a-dd68f84af8d6-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-x8s89\" (UID: \"9957ce75-f34c-42b9-804a-dd68f84af8d6\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-x8s89"
Sep 15 18:51:48.691208 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:48.691186    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9957ce75-f34c-42b9-804a-dd68f84af8d6-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-x8s89\" (UID: \"9957ce75-f34c-42b9-804a-dd68f84af8d6\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-x8s89"
Sep 15 18:51:48.880414 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:48.880376    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-x8s89"
Sep 15 18:51:49.159699 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:49.159469    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-x8s89"]
Sep 15 18:51:49.164642 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:49.164595    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9957ce75_f34c_42b9_804a_dd68f84af8d6.slice/crio-daae67393f3ec75dd3fcf42bcca4f76fed09378879c3b9fb5411ef12cca283ad WatchSource:0}: Error finding container daae67393f3ec75dd3fcf42bcca4f76fed09378879c3b9fb5411ef12cca283ad: Status 404 returned error can't find the container with id daae67393f3ec75dd3fcf42bcca4f76fed09378879c3b9fb5411ef12cca283ad
Sep 15 18:51:49.331042 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:49.331013    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zknv8" event={"ID":"9c998a93-b454-41f5-a589-87f5364c062c","Type":"ContainerStarted","Data":"4f1a1957bece3adb249bfc31c1339786007882c0a9c27aada58994b31672fec9"}
Sep 15 18:51:49.332769 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:49.332737    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-x8s89" event={"ID":"9957ce75-f34c-42b9-804a-dd68f84af8d6","Type":"ContainerStarted","Data":"daae67393f3ec75dd3fcf42bcca4f76fed09378879c3b9fb5411ef12cca283ad"}
Sep 15 18:51:49.344872 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:49.344831    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zknv8" podStartSLOduration=1.9344150469999999 podStartE2EDuration="5.344816084s" totalImagesPullingTime="3.410401037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:49.344481939 +0000 UTC m=+50.831810934" watchObservedRunningTime="2026-09-15 18:51:49.344816084 +0000 UTC m=+50.832145070"
Sep 15 18:51:50.338034 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:50.337943    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wh7k8" event={"ID":"afa9ab3a-3301-4822-b409-2cbbdb957b28","Type":"ContainerStarted","Data":"7fd0381907956c08de8fab9281a8e324c299e76ff3908851edc03a8877bb3c10"}
Sep 15 18:51:50.354875 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:50.354829    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wh7k8" podStartSLOduration=1.608530054 podStartE2EDuration="6.35481349s" totalImagesPullingTime="4.746283436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:51:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:50.353378299 +0000 UTC m=+51.840707288" watchObservedRunningTime="2026-09-15 18:51:50.35481349 +0000 UTC m=+51.842142474"
Sep 15 18:51:52.345956 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:52.345916    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-x8s89" event={"ID":"9957ce75-f34c-42b9-804a-dd68f84af8d6","Type":"ContainerStarted","Data":"9ca344aa34090af37f006f0c0df3933000d8f3f530dc18d91460884cad80e95d"}
Sep 15 18:51:52.346342 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:52.345993    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-x8s89" event={"ID":"9957ce75-f34c-42b9-804a-dd68f84af8d6","Type":"ContainerStarted","Data":"1e8058f541e74f2d07d50ace8406f1fbfa153c7f3553c4e4867357ef711ce3e6"}
Sep 15 18:51:52.361625 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:52.361565    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-x8s89" podStartSLOduration=3.148101313 podStartE2EDuration="5.361547787s" totalImagesPullingTime="2.213446474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:52.361548148 +0000 UTC m=+53.848877135" watchObservedRunningTime="2026-09-15 18:51:52.361547787 +0000 UTC m=+53.848876771"
Sep 15 18:51:54.244701 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.244670    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7"]
Sep 15 18:51:54.246757 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.246741    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7"
Sep 15 18:51:54.248985 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.248963    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 15 18:51:54.249093 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.249027    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-p6km7\""
Sep 15 18:51:54.249093 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.249028    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 15 18:51:54.256406 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.256383    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7"]
Sep 15 18:51:54.273469 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.273443    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gf7cn"]
Sep 15 18:51:54.275394 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.275375    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.276396 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.276376    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"]
Sep 15 18:51:54.277514 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.277499    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 18:51:54.277814 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.277795    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n5z5v\""
Sep 15 18:51:54.277814 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.277805    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 18:51:54.277947 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.277852    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 18:51:54.278442 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.278428    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:54.280278 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.280260    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 15 18:51:54.280960 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.280941    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 15 18:51:54.281027 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.280974    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pb9nn\""
Sep 15 18:51:54.281254 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.281240    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 15 18:51:54.292220 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.292198    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"]
Sep 15 18:51:54.334588 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.334561    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6910d9d-81c1-4eaa-975c-328baddaa833-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rd57n\" (UID: \"c6910d9d-81c1-4eaa-975c-328baddaa833\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:54.334588 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.334589    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sk8k\" (UniqueName: \"kubernetes.io/projected/c6910d9d-81c1-4eaa-975c-328baddaa833-kube-api-access-9sk8k\") pod \"kube-state-metrics-59fbdd77d-rd57n\" (UID: \"c6910d9d-81c1-4eaa-975c-328baddaa833\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:54.334775 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.334630    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-node-exporter-wtmp\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.334775 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.334681    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6910d9d-81c1-4eaa-975c-328baddaa833-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rd57n\" (UID: \"c6910d9d-81c1-4eaa-975c-328baddaa833\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:54.334775 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.334717    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-w4jn7\" (UID: \"b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7"
Sep 15 18:51:54.334775 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.334750    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c6910d9d-81c1-4eaa-975c-328baddaa833-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rd57n\" (UID: \"c6910d9d-81c1-4eaa-975c-328baddaa833\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:54.334900 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.334790    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-w4jn7\" (UID: \"b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7"
Sep 15 18:51:54.334900 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.334811    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-metrics-client-ca\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.334900 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.334835    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzmzm\" (UniqueName: \"kubernetes.io/projected/b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf-kube-api-access-gzmzm\") pod \"openshift-state-metrics-67dcf8c979-w4jn7\" (UID: \"b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7"
Sep 15 18:51:54.334900 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.334879    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-sys\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.335012 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.334907    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-node-exporter-tls\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.335012 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.334927    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6910d9d-81c1-4eaa-975c-328baddaa833-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rd57n\" (UID: \"c6910d9d-81c1-4eaa-975c-328baddaa833\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:54.335012 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.334943    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-node-exporter-textfile\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.335012 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.335000    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-node-exporter-accelerators-collector-config\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.335128 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.335053    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c6910d9d-81c1-4eaa-975c-328baddaa833-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rd57n\" (UID: \"c6910d9d-81c1-4eaa-975c-328baddaa833\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:54.335128 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.335073    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.335128 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.335087    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svg6l\" (UniqueName: \"kubernetes.io/projected/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-kube-api-access-svg6l\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.335128 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.335110    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-root\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.335243 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.335162    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-w4jn7\" (UID: \"b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7"
Sep 15 18:51:54.436029 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.435984    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c6910d9d-81c1-4eaa-975c-328baddaa833-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rd57n\" (UID: \"c6910d9d-81c1-4eaa-975c-328baddaa833\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:54.436029 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436025    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.436248 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436042    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svg6l\" (UniqueName: \"kubernetes.io/projected/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-kube-api-access-svg6l\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.436248 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436228    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-root\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.436342 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436259    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-w4jn7\" (UID: \"b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7"
Sep 15 18:51:54.436342 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436285    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6910d9d-81c1-4eaa-975c-328baddaa833-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rd57n\" (UID: \"c6910d9d-81c1-4eaa-975c-328baddaa833\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:54.436342 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436328    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-root\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.436481 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:54.436358    2555 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 15 18:51:54.436481 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436379    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sk8k\" (UniqueName: \"kubernetes.io/projected/c6910d9d-81c1-4eaa-975c-328baddaa833-kube-api-access-9sk8k\") pod \"kube-state-metrics-59fbdd77d-rd57n\" (UID: \"c6910d9d-81c1-4eaa-975c-328baddaa833\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:54.436481 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436411    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-node-exporter-wtmp\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.436481 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:54.436426    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6910d9d-81c1-4eaa-975c-328baddaa833-kube-state-metrics-tls podName:c6910d9d-81c1-4eaa-975c-328baddaa833 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:54.936406663 +0000 UTC m=+56.423735627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c6910d9d-81c1-4eaa-975c-328baddaa833-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-rd57n" (UID: "c6910d9d-81c1-4eaa-975c-328baddaa833") : secret "kube-state-metrics-tls" not found
Sep 15 18:51:54.436481 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436455    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6910d9d-81c1-4eaa-975c-328baddaa833-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rd57n\" (UID: \"c6910d9d-81c1-4eaa-975c-328baddaa833\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:54.436761 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436511    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-node-exporter-wtmp\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.436761 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436600    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-w4jn7\" (UID: \"b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7"
Sep 15 18:51:54.436761 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436672    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c6910d9d-81c1-4eaa-975c-328baddaa833-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rd57n\" (UID: \"c6910d9d-81c1-4eaa-975c-328baddaa833\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:54.436761 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436725    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-w4jn7\" (UID: \"b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7"
Sep 15 18:51:54.436761 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:54.436751    2555 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 15 18:51:54.436761 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436759    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-metrics-client-ca\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.437046 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436784    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzmzm\" (UniqueName: \"kubernetes.io/projected/b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf-kube-api-access-gzmzm\") pod \"openshift-state-metrics-67dcf8c979-w4jn7\" (UID: \"b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7"
Sep 15 18:51:54.437046 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436811    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-sys\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.437046 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:54.436828    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf-openshift-state-metrics-tls podName:b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf nodeName:}" failed. No retries permitted until 2026-09-15 18:51:54.936811277 +0000 UTC m=+56.424140244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-w4jn7" (UID: "b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf") : secret "openshift-state-metrics-tls" not found
Sep 15 18:51:54.437046 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436864    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-node-exporter-tls\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.437046 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436870    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-sys\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.437046 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436897    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6910d9d-81c1-4eaa-975c-328baddaa833-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rd57n\" (UID: \"c6910d9d-81c1-4eaa-975c-328baddaa833\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:54.437046 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436923    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-node-exporter-textfile\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.437046 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.436968    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-node-exporter-accelerators-collector-config\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.437046 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.437021    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c6910d9d-81c1-4eaa-975c-328baddaa833-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rd57n\" (UID: \"c6910d9d-81c1-4eaa-975c-328baddaa833\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:54.437439 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.437130    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-w4jn7\" (UID: \"b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7"
Sep 15 18:51:54.437439 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:54.437226    2555 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 15 18:51:54.437439 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:54.437282    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-node-exporter-tls podName:2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb nodeName:}" failed. No retries permitted until 2026-09-15 18:51:54.937265893 +0000 UTC m=+56.424594877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-node-exporter-tls") pod "node-exporter-gf7cn" (UID: "2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb") : secret "node-exporter-tls" not found
Sep 15 18:51:54.437639 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.437599    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6910d9d-81c1-4eaa-975c-328baddaa833-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rd57n\" (UID: \"c6910d9d-81c1-4eaa-975c-328baddaa833\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:54.437854 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.437834    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c6910d9d-81c1-4eaa-975c-328baddaa833-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rd57n\" (UID: \"c6910d9d-81c1-4eaa-975c-328baddaa833\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:54.437942 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.437909    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-node-exporter-textfile\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.438000 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.437961    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-metrics-client-ca\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.438048 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.438012    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-node-exporter-accelerators-collector-config\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.440314 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.440292    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.440412 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.440360    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-w4jn7\" (UID: \"b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7"
Sep 15 18:51:54.440412 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.440362    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6910d9d-81c1-4eaa-975c-328baddaa833-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rd57n\" (UID: \"c6910d9d-81c1-4eaa-975c-328baddaa833\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:54.445051 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.445013    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzmzm\" (UniqueName: \"kubernetes.io/projected/b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf-kube-api-access-gzmzm\") pod \"openshift-state-metrics-67dcf8c979-w4jn7\" (UID: \"b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7"
Sep 15 18:51:54.445149 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.445103    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svg6l\" (UniqueName: \"kubernetes.io/projected/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-kube-api-access-svg6l\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.445209 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.445188    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sk8k\" (UniqueName: \"kubernetes.io/projected/c6910d9d-81c1-4eaa-975c-328baddaa833-kube-api-access-9sk8k\") pod \"kube-state-metrics-59fbdd77d-rd57n\" (UID: \"c6910d9d-81c1-4eaa-975c-328baddaa833\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:54.940679 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.940631    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-node-exporter-tls\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.940862 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.940731    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6910d9d-81c1-4eaa-975c-328baddaa833-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rd57n\" (UID: \"c6910d9d-81c1-4eaa-975c-328baddaa833\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:54.940862 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.940766    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-w4jn7\" (UID: \"b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7"
Sep 15 18:51:54.943078 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.943045    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb-node-exporter-tls\") pod \"node-exporter-gf7cn\" (UID: \"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb\") " pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:54.943078 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.943076    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-w4jn7\" (UID: \"b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7"
Sep 15 18:51:54.943224 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:54.943208    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6910d9d-81c1-4eaa-975c-328baddaa833-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rd57n\" (UID: \"c6910d9d-81c1-4eaa-975c-328baddaa833\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:55.155796 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.155763    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7"
Sep 15 18:51:55.186386 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.186358    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gf7cn"
Sep 15 18:51:55.190794 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.190722    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"
Sep 15 18:51:55.197519 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:55.197496    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2508e5c0_9cf1_4ce6_ad7d_91f55bed0deb.slice/crio-d6773f49309f4f5e385e1037e81b132d87209bdecaf7e09fccdcece37c07e9db WatchSource:0}: Error finding container d6773f49309f4f5e385e1037e81b132d87209bdecaf7e09fccdcece37c07e9db: Status 404 returned error can't find the container with id d6773f49309f4f5e385e1037e81b132d87209bdecaf7e09fccdcece37c07e9db
Sep 15 18:51:55.329573 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.329544    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 18:51:55.335194 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.335162    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.339824 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.339748    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 15 18:51:55.339824 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.339773    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 15 18:51:55.339824 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.339783    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 15 18:51:55.339824 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.339794    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 15 18:51:55.340121 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.339793    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 15 18:51:55.340121 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.339926    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 15 18:51:55.340121 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.339938    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 15 18:51:55.340268 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.340185    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 15 18:51:55.340321 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.340299    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 15 18:51:55.340384 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.340363    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-j9kfz\""
Sep 15 18:51:55.346783 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.346764    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4411417b-070e-4696-b9ed-53d9692e16a6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.346867 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.346794    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.346867 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.346815    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4411417b-070e-4696-b9ed-53d9692e16a6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.346943 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.346865    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.346943 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.346881    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-config-volume\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.346943 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.346899    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5tlx\" (UniqueName: \"kubernetes.io/projected/4411417b-070e-4696-b9ed-53d9692e16a6-kube-api-access-m5tlx\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.347030 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.346972    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4411417b-070e-4696-b9ed-53d9692e16a6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.347030 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.346992    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.347030 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.347008    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4411417b-070e-4696-b9ed-53d9692e16a6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.347030 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.347027    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4411417b-070e-4696-b9ed-53d9692e16a6-config-out\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.347145 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.347047    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.347145 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.347072    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-web-config\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.347145 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.347099    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.359890 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.359866    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gf7cn" event={"ID":"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb","Type":"ContainerStarted","Data":"d6773f49309f4f5e385e1037e81b132d87209bdecaf7e09fccdcece37c07e9db"}
Sep 15 18:51:55.373940 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.373913    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 18:51:55.428556 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.428528    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7"]
Sep 15 18:51:55.432078 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:55.432054    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ae0cf0_f0a5_428d_82bf_0509b4d94dcf.slice/crio-d27c641f039f2f62bf6fe51dcf12afda2fd66e2505e39ce41eabeb84e086daba WatchSource:0}: Error finding container d27c641f039f2f62bf6fe51dcf12afda2fd66e2505e39ce41eabeb84e086daba: Status 404 returned error can't find the container with id d27c641f039f2f62bf6fe51dcf12afda2fd66e2505e39ce41eabeb84e086daba
Sep 15 18:51:55.447659 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.447602    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4411417b-070e-4696-b9ed-53d9692e16a6-config-out\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.447810 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.447673    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.447810 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.447702    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-web-config\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.447810 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.447739    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.447810 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.447770    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4411417b-070e-4696-b9ed-53d9692e16a6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.447810 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.447802    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.448069 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.447828    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4411417b-070e-4696-b9ed-53d9692e16a6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.448252 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.448211    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.448491 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.448446    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-config-volume\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.448717 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:55.448548    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4411417b-070e-4696-b9ed-53d9692e16a6-alertmanager-trusted-ca-bundle podName:4411417b-070e-4696-b9ed-53d9692e16a6 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:55.948524256 +0000 UTC m=+57.435853220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/4411417b-070e-4696-b9ed-53d9692e16a6-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "4411417b-070e-4696-b9ed-53d9692e16a6") : configmap references non-existent config key: ca-bundle.crt
Sep 15 18:51:55.448981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.448814    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5tlx\" (UniqueName: \"kubernetes.io/projected/4411417b-070e-4696-b9ed-53d9692e16a6-kube-api-access-m5tlx\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.449232 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.449070    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4411417b-070e-4696-b9ed-53d9692e16a6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.449518 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.449265    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4411417b-070e-4696-b9ed-53d9692e16a6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.449518 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.449286    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.449518 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.449316    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4411417b-070e-4696-b9ed-53d9692e16a6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.450112 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.450091    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4411417b-070e-4696-b9ed-53d9692e16a6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.450589 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.450566    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4411417b-070e-4696-b9ed-53d9692e16a6-config-out\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.452279 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.451856    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4411417b-070e-4696-b9ed-53d9692e16a6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.452279 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.452219    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-web-config\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.452955 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.452915    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.454011 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.453746    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.454011 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.453973    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.455883 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.454404    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-config-volume\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.456162 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.456121    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.456257 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.456198    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.458157 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.458126    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5tlx\" (UniqueName: \"kubernetes.io/projected/4411417b-070e-4696-b9ed-53d9692e16a6-kube-api-access-m5tlx\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.470584 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.470561    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n"]
Sep 15 18:51:55.473722 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:55.473700    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6910d9d_81c1_4eaa_975c_328baddaa833.slice/crio-13838720d399d957091befa8c6d5546a65c4d1cb00d9a3cb5beb14571099663e WatchSource:0}: Error finding container 13838720d399d957091befa8c6d5546a65c4d1cb00d9a3cb5beb14571099663e: Status 404 returned error can't find the container with id 13838720d399d957091befa8c6d5546a65c4d1cb00d9a3cb5beb14571099663e
Sep 15 18:51:55.953712 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.953673    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4411417b-070e-4696-b9ed-53d9692e16a6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:55.954414 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:55.954385    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4411417b-070e-4696-b9ed-53d9692e16a6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:56.246117 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:56.246083    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:51:56.368943 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:56.368664    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n" event={"ID":"c6910d9d-81c1-4eaa-975c-328baddaa833","Type":"ContainerStarted","Data":"13838720d399d957091befa8c6d5546a65c4d1cb00d9a3cb5beb14571099663e"}
Sep 15 18:51:56.372385 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:56.372254    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7" event={"ID":"b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf","Type":"ContainerStarted","Data":"fa09d3efe5c3841fd5fbf37b7f241b7071d318342dad786edf40dc84a8a33269"}
Sep 15 18:51:56.372385 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:56.372316    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7" event={"ID":"b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf","Type":"ContainerStarted","Data":"69a19ed8ac2d4e98f8f8b39bf1da75fe8cd2c1cd3ced1d5b792551a0a7551619"}
Sep 15 18:51:56.372385 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:56.372346    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7" event={"ID":"b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf","Type":"ContainerStarted","Data":"d27c641f039f2f62bf6fe51dcf12afda2fd66e2505e39ce41eabeb84e086daba"}
Sep 15 18:51:56.599080 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:56.599055    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 18:51:56.833478 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:56.833442    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4411417b_070e_4696_b9ed_53d9692e16a6.slice/crio-a931b2a413fc8623f4770bc696ee1d4ab60250d4e4356c2f5bb3764dfb1366d0 WatchSource:0}: Error finding container a931b2a413fc8623f4770bc696ee1d4ab60250d4e4356c2f5bb3764dfb1366d0: Status 404 returned error can't find the container with id a931b2a413fc8623f4770bc696ee1d4ab60250d4e4356c2f5bb3764dfb1366d0
Sep 15 18:51:57.239776 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.239693    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5d94699b48-nfzfw"]
Sep 15 18:51:57.247350 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.247325    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.249910 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.249696    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 15 18:51:57.249910 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.249808    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 15 18:51:57.249910 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.249852    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 15 18:51:57.249910 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.249845    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-837f7tiljeoql\""
Sep 15 18:51:57.250184 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.249958    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 15 18:51:57.250184 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.250023    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 15 18:51:57.250278 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.250209    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vwwpc\""
Sep 15 18:51:57.254578 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.254556    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d94699b48-nfzfw"]
Sep 15 18:51:57.264768 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.264743    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9769660-5ff3-42a3-8df2-9202ffce4803-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.264904 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.264773    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9769660-5ff3-42a3-8df2-9202ffce4803-metrics-client-ca\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.264904 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.264889    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9769660-5ff3-42a3-8df2-9202ffce4803-secret-grpc-tls\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.265014 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.264924    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ql7b\" (UniqueName: \"kubernetes.io/projected/c9769660-5ff3-42a3-8df2-9202ffce4803-kube-api-access-9ql7b\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.265014 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.264969    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9769660-5ff3-42a3-8df2-9202ffce4803-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.265124 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.265020    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9769660-5ff3-42a3-8df2-9202ffce4803-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.265124 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.265045    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9769660-5ff3-42a3-8df2-9202ffce4803-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.265124 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.265065    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9769660-5ff3-42a3-8df2-9202ffce4803-secret-thanos-querier-tls\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.365845 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.365808    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9769660-5ff3-42a3-8df2-9202ffce4803-secret-thanos-querier-tls\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.366024 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.365874    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9769660-5ff3-42a3-8df2-9202ffce4803-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.366024 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.365895    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9769660-5ff3-42a3-8df2-9202ffce4803-metrics-client-ca\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.366024 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.365942    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9769660-5ff3-42a3-8df2-9202ffce4803-secret-grpc-tls\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.366024 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.365956    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ql7b\" (UniqueName: \"kubernetes.io/projected/c9769660-5ff3-42a3-8df2-9202ffce4803-kube-api-access-9ql7b\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.366024 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.365983    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9769660-5ff3-42a3-8df2-9202ffce4803-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.366024 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.366009    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9769660-5ff3-42a3-8df2-9202ffce4803-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.366322 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.366037    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9769660-5ff3-42a3-8df2-9202ffce4803-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.366857 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.366800    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9769660-5ff3-42a3-8df2-9202ffce4803-metrics-client-ca\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.368657 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.368591    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9769660-5ff3-42a3-8df2-9202ffce4803-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.368796 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.368774    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9769660-5ff3-42a3-8df2-9202ffce4803-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.368975 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.368955    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9769660-5ff3-42a3-8df2-9202ffce4803-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.369274 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.368989    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9769660-5ff3-42a3-8df2-9202ffce4803-secret-thanos-querier-tls\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.369476 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.369456    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9769660-5ff3-42a3-8df2-9202ffce4803-secret-grpc-tls\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.369591 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.369575    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9769660-5ff3-42a3-8df2-9202ffce4803-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.374713 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.374691    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ql7b\" (UniqueName: \"kubernetes.io/projected/c9769660-5ff3-42a3-8df2-9202ffce4803-kube-api-access-9ql7b\") pod \"thanos-querier-5d94699b48-nfzfw\" (UID: \"c9769660-5ff3-42a3-8df2-9202ffce4803\") " pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.376620 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.376585    2555 generic.go:356] "Generic (PLEG): container finished" podID="2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb" containerID="e1da82be802240552565a9783d4361cd0c00cc2e4541541a4e77c2613cf1175c" exitCode=0
Sep 15 18:51:57.376718 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.376643    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gf7cn" event={"ID":"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb","Type":"ContainerDied","Data":"e1da82be802240552565a9783d4361cd0c00cc2e4541541a4e77c2613cf1175c"}
Sep 15 18:51:57.378969 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.378948    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n" event={"ID":"c6910d9d-81c1-4eaa-975c-328baddaa833","Type":"ContainerStarted","Data":"69d379ddb012da340ba9d8dd432025eb938feca087d2112c4782559a4d0c9e15"}
Sep 15 18:51:57.379071 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.379002    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n" event={"ID":"c6910d9d-81c1-4eaa-975c-328baddaa833","Type":"ContainerStarted","Data":"e27a6e5ae0778e42e25022d800791a48fdf3e0f73dcc531168ae8175c2a37f79"}
Sep 15 18:51:57.379071 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.379030    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n" event={"ID":"c6910d9d-81c1-4eaa-975c-328baddaa833","Type":"ContainerStarted","Data":"6f5298b8f5a2e4a6cd18d726d1c3d6ed52692fa36729617f6eff9394bf8015a6"}
Sep 15 18:51:57.380202 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.380186    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4411417b-070e-4696-b9ed-53d9692e16a6","Type":"ContainerStarted","Data":"a931b2a413fc8623f4770bc696ee1d4ab60250d4e4356c2f5bb3764dfb1366d0"}
Sep 15 18:51:57.408309 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.408266    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rd57n" podStartSLOduration=2.026263028 podStartE2EDuration="3.408251566s" totalImagesPullingTime="1.381988538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:57.40723471 +0000 UTC m=+58.894563694" watchObservedRunningTime="2026-09-15 18:51:57.408251566 +0000 UTC m=+58.895580551"
Sep 15 18:51:57.557835 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.557791    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:51:57.872386 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:57.872364    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d94699b48-nfzfw"]
Sep 15 18:51:57.874765 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:57.874739    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9769660_5ff3_42a3_8df2_9202ffce4803.slice/crio-6fb273b85f37a2d340bd936881a534f0cdc6e8ad04c5ba5925756c599cf0c339 WatchSource:0}: Error finding container 6fb273b85f37a2d340bd936881a534f0cdc6e8ad04c5ba5925756c599cf0c339: Status 404 returned error can't find the container with id 6fb273b85f37a2d340bd936881a534f0cdc6e8ad04c5ba5925756c599cf0c339
Sep 15 18:51:58.335781 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.335747    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rldqf"
Sep 15 18:51:58.387012 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.386962    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7" event={"ID":"b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf","Type":"ContainerStarted","Data":"33c0561c3b0ca1cc4c9d960a445b79cac3a28b2196c5848da36d652fc368afe3"}
Sep 15 18:51:58.388468 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.388438    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw" event={"ID":"c9769660-5ff3-42a3-8df2-9202ffce4803","Type":"ContainerStarted","Data":"6fb273b85f37a2d340bd936881a534f0cdc6e8ad04c5ba5925756c599cf0c339"}
Sep 15 18:51:58.391073 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.391047    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gf7cn" event={"ID":"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb","Type":"ContainerStarted","Data":"c4e6a099e3ebe774c44cc1947a332b2618c77a448758a995d0da288b39bf26b3"}
Sep 15 18:51:58.391186 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.391099    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gf7cn" event={"ID":"2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb","Type":"ContainerStarted","Data":"816ebfaf36b135a9154308ed0eeb536db40dc8f8c0aabf0f6f8ef33bd6bd8632"}
Sep 15 18:51:58.411739 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.410629    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-w4jn7" podStartSLOduration=2.237725626 podStartE2EDuration="4.41059714s" totalImagesPullingTime="2.172871514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:58.408583938 +0000 UTC m=+59.895912925" watchObservedRunningTime="2026-09-15 18:51:58.41059714 +0000 UTC m=+59.897926126"
Sep 15 18:51:58.427790 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.427746    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gf7cn" podStartSLOduration=3.215593214 podStartE2EDuration="4.427731821s" totalImagesPullingTime="1.142558046s" totalInitContainerRuntime="69.580561ms" isStatefulPod=true podCreationTimestamp="2026-09-15 18:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:58.426961567 +0000 UTC m=+59.914290563" watchObservedRunningTime="2026-09-15 18:51:58.427731821 +0000 UTC m=+59.915060806"
Sep 15 18:51:58.566779 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.566732    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"]
Sep 15 18:51:58.581447 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.581420    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"]
Sep 15 18:51:58.581588 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.581523    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.583866 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.583840    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 15 18:51:58.584005 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.583875    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 15 18:51:58.584005 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.583875    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 15 18:51:58.584005 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.583928    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-cxsjs\""
Sep 15 18:51:58.584005 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.583878    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4a75lhih7usiv\""
Sep 15 18:51:58.584005 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.583977    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 15 18:51:58.679583 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.679544    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d711a256-d08f-488a-ac0a-4950a9e7101e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.679583 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.679583    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d711a256-d08f-488a-ac0a-4950a9e7101e-metrics-server-audit-profiles\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.679801 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.679692    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d711a256-d08f-488a-ac0a-4950a9e7101e-audit-log\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.679801 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.679735    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d711a256-d08f-488a-ac0a-4950a9e7101e-secret-metrics-server-tls\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.679801 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.679790    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d711a256-d08f-488a-ac0a-4950a9e7101e-secret-metrics-server-client-certs\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.679910 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.679807    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d711a256-d08f-488a-ac0a-4950a9e7101e-client-ca-bundle\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.679910 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.679834    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2pn2\" (UniqueName: \"kubernetes.io/projected/d711a256-d08f-488a-ac0a-4950a9e7101e-kube-api-access-d2pn2\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.780921 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.780882    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d711a256-d08f-488a-ac0a-4950a9e7101e-secret-metrics-server-tls\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.781095 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.780954    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d711a256-d08f-488a-ac0a-4950a9e7101e-secret-metrics-server-client-certs\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.781095 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.780981    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d711a256-d08f-488a-ac0a-4950a9e7101e-client-ca-bundle\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.781095 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.781020    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2pn2\" (UniqueName: \"kubernetes.io/projected/d711a256-d08f-488a-ac0a-4950a9e7101e-kube-api-access-d2pn2\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.781095 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.781052    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d711a256-d08f-488a-ac0a-4950a9e7101e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.781095 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.781089    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d711a256-d08f-488a-ac0a-4950a9e7101e-metrics-server-audit-profiles\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.781361 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.781275    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d711a256-d08f-488a-ac0a-4950a9e7101e-audit-log\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.781688 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.781662    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d711a256-d08f-488a-ac0a-4950a9e7101e-audit-log\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.781829 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.781806    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d711a256-d08f-488a-ac0a-4950a9e7101e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.782168 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.782148    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d711a256-d08f-488a-ac0a-4950a9e7101e-metrics-server-audit-profiles\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.783503 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.783476    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d711a256-d08f-488a-ac0a-4950a9e7101e-secret-metrics-server-tls\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.783936 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.783915    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d711a256-d08f-488a-ac0a-4950a9e7101e-client-ca-bundle\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.783984 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.783954    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d711a256-d08f-488a-ac0a-4950a9e7101e-secret-metrics-server-client-certs\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.788378 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.788356    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2pn2\" (UniqueName: \"kubernetes.io/projected/d711a256-d08f-488a-ac0a-4950a9e7101e-kube-api-access-d2pn2\") pod \"metrics-server-57bbd7bfc6-b6nqj\" (UID: \"d711a256-d08f-488a-ac0a-4950a9e7101e\") " pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:58.893766 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.893747    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-cxsjs\""
Sep 15 18:51:58.902433 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:58.902406    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:51:59.032562 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:59.032537    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-bqdnz"]
Sep 15 18:51:59.051376 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:59.051349    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bqdnz"
Sep 15 18:51:59.054198 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:59.053954    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 15 18:51:59.054198 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:59.054016    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-sxnz7\""
Sep 15 18:51:59.063324 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:59.063304    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-bqdnz"]
Sep 15 18:51:59.084125 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:59.084099    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24w98\" (UniqueName: \"kubernetes.io/projected/c5c66802-2cb3-4202-9ed5-f46fe621aa2b-kube-api-access-24w98\") pod \"monitoring-plugin-5d85778b9d-bqdnz\" (UID: \"c5c66802-2cb3-4202-9ed5-f46fe621aa2b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bqdnz"
Sep 15 18:51:59.084245 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:59.084224    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c5c66802-2cb3-4202-9ed5-f46fe621aa2b-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-bqdnz\" (UID: \"c5c66802-2cb3-4202-9ed5-f46fe621aa2b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bqdnz"
Sep 15 18:51:59.086814 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:59.086792    2555 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4411417b_070e_4696_b9ed_53d9692e16a6.slice/crio-conmon-881182aee9f095367413589e18c33a735f4d3322eddde72136bc835e9836c7c0.scope\": RecentStats: unable to find data in memory cache]"
Sep 15 18:51:59.185778 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:59.185680    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24w98\" (UniqueName: \"kubernetes.io/projected/c5c66802-2cb3-4202-9ed5-f46fe621aa2b-kube-api-access-24w98\") pod \"monitoring-plugin-5d85778b9d-bqdnz\" (UID: \"c5c66802-2cb3-4202-9ed5-f46fe621aa2b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bqdnz"
Sep 15 18:51:59.185923 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:59.185785    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c5c66802-2cb3-4202-9ed5-f46fe621aa2b-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-bqdnz\" (UID: \"c5c66802-2cb3-4202-9ed5-f46fe621aa2b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bqdnz"
Sep 15 18:51:59.185923 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:59.185906    2555 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 15 18:51:59.186011 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:51:59.185975    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5c66802-2cb3-4202-9ed5-f46fe621aa2b-monitoring-plugin-cert podName:c5c66802-2cb3-4202-9ed5-f46fe621aa2b nodeName:}" failed. No retries permitted until 2026-09-15 18:51:59.685954961 +0000 UTC m=+61.173283925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/c5c66802-2cb3-4202-9ed5-f46fe621aa2b-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-bqdnz" (UID: "c5c66802-2cb3-4202-9ed5-f46fe621aa2b") : secret "monitoring-plugin-cert" not found
Sep 15 18:51:59.196455 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:59.196429    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24w98\" (UniqueName: \"kubernetes.io/projected/c5c66802-2cb3-4202-9ed5-f46fe621aa2b-kube-api-access-24w98\") pod \"monitoring-plugin-5d85778b9d-bqdnz\" (UID: \"c5c66802-2cb3-4202-9ed5-f46fe621aa2b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bqdnz"
Sep 15 18:51:59.207368 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:59.207334    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"]
Sep 15 18:51:59.210974 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:51:59.210939    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd711a256_d08f_488a_ac0a_4950a9e7101e.slice/crio-7a6613a125bbd353850f2c5f11cb29164046f702aaedff4d5ac41308763f078d WatchSource:0}: Error finding container 7a6613a125bbd353850f2c5f11cb29164046f702aaedff4d5ac41308763f078d: Status 404 returned error can't find the container with id 7a6613a125bbd353850f2c5f11cb29164046f702aaedff4d5ac41308763f078d
Sep 15 18:51:59.273400 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:59.273373    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cfcxz"
Sep 15 18:51:59.396158 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:59.396122    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj" event={"ID":"d711a256-d08f-488a-ac0a-4950a9e7101e","Type":"ContainerStarted","Data":"7a6613a125bbd353850f2c5f11cb29164046f702aaedff4d5ac41308763f078d"}
Sep 15 18:51:59.397999 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:59.397973    2555 generic.go:356] "Generic (PLEG): container finished" podID="4411417b-070e-4696-b9ed-53d9692e16a6" containerID="881182aee9f095367413589e18c33a735f4d3322eddde72136bc835e9836c7c0" exitCode=0
Sep 15 18:51:59.398123 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:59.398060    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4411417b-070e-4696-b9ed-53d9692e16a6","Type":"ContainerDied","Data":"881182aee9f095367413589e18c33a735f4d3322eddde72136bc835e9836c7c0"}
Sep 15 18:51:59.691947 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:59.691911    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c5c66802-2cb3-4202-9ed5-f46fe621aa2b-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-bqdnz\" (UID: \"c5c66802-2cb3-4202-9ed5-f46fe621aa2b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bqdnz"
Sep 15 18:51:59.694478 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:59.694457    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c5c66802-2cb3-4202-9ed5-f46fe621aa2b-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-bqdnz\" (UID: \"c5c66802-2cb3-4202-9ed5-f46fe621aa2b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bqdnz"
Sep 15 18:51:59.964842 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:51:59.964549    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bqdnz"
Sep 15 18:52:00.515644 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.515302    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 18:52:00.521297 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.521271    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.523959 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.523897    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 18:52:00.524939 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.524917    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 18:52:00.525166 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.525148    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 18:52:00.525353 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.525337    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 18:52:00.526804 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.526783    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 18:52:00.527159 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.527143    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 18:52:00.527326 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.527302    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 18:52:00.527482 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.527465    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 18:52:00.527563 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.527550    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bcfdz\""
Sep 15 18:52:00.527832 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.527816    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 18:52:00.528022 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.527995    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 18:52:00.528197 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.528181    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 18:52:00.528435 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.528403    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3eiolerg054sr\""
Sep 15 18:52:00.528880 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.528861    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 18:52:00.551115 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.544201    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 18:52:00.551115 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.546920    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 18:52:00.555251 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.555224    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 18:52:00.602515 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.602473    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.602714 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.602539    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.602714 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.602570    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.602714 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.602646    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.602714 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.602673    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.602714 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.602696    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.602985 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.602725    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.602985 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.602749    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlgpq\" (UniqueName: \"kubernetes.io/projected/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-kube-api-access-zlgpq\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.602985 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.602788    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-web-config\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.602985 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.602820    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.602985 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.602842    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.602985 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.602869    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-config-out\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.602985 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.602891    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.602985 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.602915    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-config\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.602985 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.602946    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.602985 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.602968    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.603464 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.603001    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.603464 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.603058    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.603464 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.603098    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.603464 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.603126    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704019 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.703875    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704019 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.703931    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704019 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.703976    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704019 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.704024    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704357 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.704049    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704357 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.704098    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704357 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.704119    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704357 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.704141    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704357 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.704166    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704357 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.704192    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlgpq\" (UniqueName: \"kubernetes.io/projected/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-kube-api-access-zlgpq\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704357 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.704240    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-web-config\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704357 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.704272    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704357 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.704296    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704357 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.704322    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-config-out\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704930 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.704387    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704930 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.704415    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-config\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704930 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.704443    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704930 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.704468    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704930 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.704500    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704930 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.704555    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.704930 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.704884    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.706343 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.706245    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.706983 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.706955    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.709872 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.708314    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.709872 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.708598    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.709872 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.708905    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.709872 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.709546    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.711838 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.711459    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.711950 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.711870    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.720171 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.720145    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-config-out\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.720283 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.720229    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlgpq\" (UniqueName: \"kubernetes.io/projected/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-kube-api-access-zlgpq\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.720557 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.720534    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.720788 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.720764    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.720788 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.720778    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-web-config\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.721460 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.721435    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.721549 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.721532    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.721884 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.721864    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.722031 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.721909    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-config\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.724907 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.724182    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.726626 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.726590    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:00.851777 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:00.851566    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:01.010421 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:01.009986    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-bqdnz"]
Sep 15 18:52:01.243407 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:52:01.243379    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c66802_2cb3_4202_9ed5_f46fe621aa2b.slice/crio-c5d210f398a0fc71e4efb4164607428d00e4f944b23896093d3d773183f4fe42 WatchSource:0}: Error finding container c5d210f398a0fc71e4efb4164607428d00e4f944b23896093d3d773183f4fe42: Status 404 returned error can't find the container with id c5d210f398a0fc71e4efb4164607428d00e4f944b23896093d3d773183f4fe42
Sep 15 18:52:01.409120 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:01.409070    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw" event={"ID":"c9769660-5ff3-42a3-8df2-9202ffce4803","Type":"ContainerStarted","Data":"815ffb241097c1e28ddbdfe8cab23c982253083c8cb193c350bf70da357efdca"}
Sep 15 18:52:01.410551 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:01.410520    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bqdnz" event={"ID":"c5c66802-2cb3-4202-9ed5-f46fe621aa2b","Type":"ContainerStarted","Data":"c5d210f398a0fc71e4efb4164607428d00e4f944b23896093d3d773183f4fe42"}
Sep 15 18:52:01.875090 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:01.874813    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 18:52:02.417775 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:02.417736    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw" event={"ID":"c9769660-5ff3-42a3-8df2-9202ffce4803","Type":"ContainerStarted","Data":"d594148e0fb7234c039bb7fcfb2fb48b54a3059ce0b233237220fdc2e7c8bbc3"}
Sep 15 18:52:02.417947 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:02.417855    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw" event={"ID":"c9769660-5ff3-42a3-8df2-9202ffce4803","Type":"ContainerStarted","Data":"0e81d9e0c93186a2dc32fefaee88a21fc445d8edbce302ab5ce68d7f307fc85d"}
Sep 15 18:52:02.419594 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:02.419567    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj" event={"ID":"d711a256-d08f-488a-ac0a-4950a9e7101e","Type":"ContainerStarted","Data":"946f4da582e602e828b1b2e6dfac16e2085eb939f714bdc4cc442ce434ddc614"}
Sep 15 18:52:02.423326 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:02.423299    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4411417b-070e-4696-b9ed-53d9692e16a6","Type":"ContainerStarted","Data":"0ebfea9981ae1f7386f2881473362e37352c1f150b701f79c60b4732dc855e0a"}
Sep 15 18:52:02.423413 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:02.423361    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4411417b-070e-4696-b9ed-53d9692e16a6","Type":"ContainerStarted","Data":"35827e8da70aa5d24f59ce321828e241c27d945a78f3a92851c93665bdf3f618"}
Sep 15 18:52:02.423413 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:02.423395    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4411417b-070e-4696-b9ed-53d9692e16a6","Type":"ContainerStarted","Data":"43db3a3ccdc557d4b83c5bb3fb51cbf0e107575eeafe9f68adc1de0f5b994d19"}
Sep 15 18:52:02.423515 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:02.423426    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4411417b-070e-4696-b9ed-53d9692e16a6","Type":"ContainerStarted","Data":"c824620aa507f06b955e59c3b4e7af828c089e7b65def8ebf3004a3be22da99e"}
Sep 15 18:52:02.423515 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:02.423465    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4411417b-070e-4696-b9ed-53d9692e16a6","Type":"ContainerStarted","Data":"08d75c28331e827e49da7e92c09d3b25336cbab9380f93741c37a1762b523c45"}
Sep 15 18:52:02.425039 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:02.425014    2555 generic.go:356] "Generic (PLEG): container finished" podID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerID="6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406" exitCode=0
Sep 15 18:52:02.425141 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:02.425086    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41","Type":"ContainerDied","Data":"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406"}
Sep 15 18:52:02.425207 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:02.425161    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41","Type":"ContainerStarted","Data":"3f907ca59b197e5a5148f98b53526e0c5d7ff6aab80e037246763a15835ad072"}
Sep 15 18:52:02.436445 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:02.436398    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj" podStartSLOduration=2.053376344 podStartE2EDuration="4.436380554s" totalImagesPullingTime="2.38300421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:52:02.434038175 +0000 UTC m=+63.921367161" watchObservedRunningTime="2026-09-15 18:52:02.436380554 +0000 UTC m=+63.923709626"
Sep 15 18:52:03.434708 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:03.434669    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw" event={"ID":"c9769660-5ff3-42a3-8df2-9202ffce4803","Type":"ContainerStarted","Data":"1ce06ccad61302e99b2a301f83aade654ee63722ec4802ffd146b3c7b9bb9389"}
Sep 15 18:52:03.435126 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:03.434791    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw" event={"ID":"c9769660-5ff3-42a3-8df2-9202ffce4803","Type":"ContainerStarted","Data":"93f604caad726dca8e4a3eecd45f137bc919161230f8a12a09ef2ceb35c302ff"}
Sep 15 18:52:03.435126 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:03.434833    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw" event={"ID":"c9769660-5ff3-42a3-8df2-9202ffce4803","Type":"ContainerStarted","Data":"113f5eb5faba87b1b37384d31a247b46d435a978cc9e87eeabdb31820dc41208"}
Sep 15 18:52:03.435126 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:03.434899    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:52:03.437039 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:03.437012    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bqdnz" event={"ID":"c5c66802-2cb3-4202-9ed5-f46fe621aa2b","Type":"ContainerStarted","Data":"926638ff35ac0b2d69cb66888a8faed8e2df9716d132f7e09d9bdbe6928227ce"}
Sep 15 18:52:03.437445 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:03.437405    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bqdnz"
Sep 15 18:52:03.442893 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:03.442868    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4411417b-070e-4696-b9ed-53d9692e16a6","Type":"ContainerStarted","Data":"b25a94eb1fbf499e7f98914b776ba5a51c7a5ef41f7ec3d7c650e468e72a3e3f"}
Sep 15 18:52:03.444487 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:03.444467    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bqdnz"
Sep 15 18:52:03.456218 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:03.456133    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw" podStartSLOduration=2.098409394 podStartE2EDuration="6.456114566s" totalImagesPullingTime="4.357705172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:52:03.454412721 +0000 UTC m=+64.941741718" watchObservedRunningTime="2026-09-15 18:52:03.456114566 +0000 UTC m=+64.943443554"
Sep 15 18:52:03.467848 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:03.467802    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bqdnz" podStartSLOduration=3.033147355 podStartE2EDuration="4.467787222s" totalImagesPullingTime="1.434639867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:52:03.466926498 +0000 UTC m=+64.954255494" watchObservedRunningTime="2026-09-15 18:52:03.467787222 +0000 UTC m=+64.955116206"
Sep 15 18:52:03.490012 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:03.489967    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.956607342 podStartE2EDuration="8.489951601s" totalImagesPullingTime="5.514896134s" totalInitContainerRuntime="18.448125ms" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:52:03.48822847 +0000 UTC m=+64.975557454" watchObservedRunningTime="2026-09-15 18:52:03.489951601 +0000 UTC m=+64.977280585"
Sep 15 18:52:04.747011 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:04.746969    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret\") pod \"global-pull-secret-syncer-j8kt6\" (UID: \"f16c1599-2406-48bc-88fb-36e9670dc851\") " pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:52:04.749219 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:04.749196    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 18:52:04.760012 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:04.759988    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f16c1599-2406-48bc-88fb-36e9670dc851-original-pull-secret\") pod \"global-pull-secret-syncer-j8kt6\" (UID: \"f16c1599-2406-48bc-88fb-36e9670dc851\") " pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:52:04.847737 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:04.847699    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8j6\" (UniqueName: \"kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6\") pod \"network-check-target-wxhxs\" (UID: \"f9e0b82d-6132-4018-9b73-cb4bb6639ba2\") " pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:52:04.847903 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:04.847792    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs\") pod \"network-metrics-daemon-lpczh\" (UID: \"0a36af7f-abf7-414c-80b2-6314083e1a60\") " pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:52:04.849941 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:04.849882    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 18:52:04.850067 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:04.849953    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 18:52:04.860887 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:04.860860    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 18:52:04.861023 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:04.860978    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a36af7f-abf7-414c-80b2-6314083e1a60-metrics-certs\") pod \"network-metrics-daemon-lpczh\" (UID: \"0a36af7f-abf7-414c-80b2-6314083e1a60\") " pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:52:04.871906 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:04.871884    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c8j6\" (UniqueName: \"kubernetes.io/projected/f9e0b82d-6132-4018-9b73-cb4bb6639ba2-kube-api-access-2c8j6\") pod \"network-check-target-wxhxs\" (UID: \"f9e0b82d-6132-4018-9b73-cb4bb6639ba2\") " pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:52:04.947867 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:04.947830    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4h8bk\""
Sep 15 18:52:04.953913 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:04.953883    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8kt6"
Sep 15 18:52:04.956007 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:04.955980    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:52:04.961310 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:04.961282    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gnvqj\""
Sep 15 18:52:04.969282 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:04.969256    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lpczh"
Sep 15 18:52:06.266396 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:06.266332    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j8kt6"]
Sep 15 18:52:06.269651 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:06.269594    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wxhxs"]
Sep 15 18:52:06.270457 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:52:06.270427    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16c1599_2406_48bc_88fb_36e9670dc851.slice/crio-305ffecaf7b98c1ff6f2f3debb613742929963008223b05b128c166cc524b138 WatchSource:0}: Error finding container 305ffecaf7b98c1ff6f2f3debb613742929963008223b05b128c166cc524b138: Status 404 returned error can't find the container with id 305ffecaf7b98c1ff6f2f3debb613742929963008223b05b128c166cc524b138
Sep 15 18:52:06.273276 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:52:06.273243    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e0b82d_6132_4018_9b73_cb4bb6639ba2.slice/crio-b46d8e22fc14d2e075150a672ec57694d5eec73c60f82008ac96ca3978c1990d WatchSource:0}: Error finding container b46d8e22fc14d2e075150a672ec57694d5eec73c60f82008ac96ca3978c1990d: Status 404 returned error can't find the container with id b46d8e22fc14d2e075150a672ec57694d5eec73c60f82008ac96ca3978c1990d
Sep 15 18:52:06.279175 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:06.279153    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lpczh"]
Sep 15 18:52:06.280967 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:52:06.280944    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a36af7f_abf7_414c_80b2_6314083e1a60.slice/crio-d9232ba13947da0e33d3140b10bdaecbbb06740d8e3957a79a1c863368e09f1a WatchSource:0}: Error finding container d9232ba13947da0e33d3140b10bdaecbbb06740d8e3957a79a1c863368e09f1a: Status 404 returned error can't find the container with id d9232ba13947da0e33d3140b10bdaecbbb06740d8e3957a79a1c863368e09f1a
Sep 15 18:52:06.459801 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:06.459770    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41","Type":"ContainerStarted","Data":"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6"}
Sep 15 18:52:06.459915 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:06.459877    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41","Type":"ContainerStarted","Data":"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda"}
Sep 15 18:52:06.460037 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:06.459936    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41","Type":"ContainerStarted","Data":"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2"}
Sep 15 18:52:06.462121 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:06.462090    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lpczh" event={"ID":"0a36af7f-abf7-414c-80b2-6314083e1a60","Type":"ContainerStarted","Data":"d9232ba13947da0e33d3140b10bdaecbbb06740d8e3957a79a1c863368e09f1a"}
Sep 15 18:52:06.463978 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:06.463942    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wxhxs" event={"ID":"f9e0b82d-6132-4018-9b73-cb4bb6639ba2","Type":"ContainerStarted","Data":"b46d8e22fc14d2e075150a672ec57694d5eec73c60f82008ac96ca3978c1990d"}
Sep 15 18:52:06.465399 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:06.465374    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j8kt6" event={"ID":"f16c1599-2406-48bc-88fb-36e9670dc851","Type":"ContainerStarted","Data":"305ffecaf7b98c1ff6f2f3debb613742929963008223b05b128c166cc524b138"}
Sep 15 18:52:07.479354 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:07.479312    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41","Type":"ContainerStarted","Data":"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe"}
Sep 15 18:52:07.479816 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:07.479409    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41","Type":"ContainerStarted","Data":"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9"}
Sep 15 18:52:07.479816 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:07.479455    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41","Type":"ContainerStarted","Data":"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb"}
Sep 15 18:52:07.509157 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:07.509086    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.734708598 podStartE2EDuration="7.509061958s" totalImagesPullingTime="3.749086914s" totalInitContainerRuntime="25.266446ms" isStatefulPod=false podCreationTimestamp="2026-09-15 18:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:52:07.507805566 +0000 UTC m=+68.995134563" watchObservedRunningTime="2026-09-15 18:52:07.509061958 +0000 UTC m=+68.996390944"
Sep 15 18:52:08.486847 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:08.486736    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lpczh" event={"ID":"0a36af7f-abf7-414c-80b2-6314083e1a60","Type":"ContainerStarted","Data":"ec22142fc4420348a29e144093f6321ab52e92fb030bc8026dc96846e6442c50"}
Sep 15 18:52:08.486847 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:08.486847    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lpczh" event={"ID":"0a36af7f-abf7-414c-80b2-6314083e1a60","Type":"ContainerStarted","Data":"e2864ccc481d9912c01ab75a3a971a9e7e6f64060ac9ee87fa403a9622870d69"}
Sep 15 18:52:08.502147 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:08.502081    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lpczh" podStartSLOduration=67.214284126 podStartE2EDuration="1m8.502058065s" totalImagesPullingTime="1.287773939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:52:08.501948646 +0000 UTC m=+69.989277632" watchObservedRunningTime="2026-09-15 18:52:08.502058065 +0000 UTC m=+69.989387052"
Sep 15 18:52:09.452881 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:09.452854    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5d94699b48-nfzfw"
Sep 15 18:52:10.852565 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:10.852528    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:52:11.503838 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:11.503804    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wxhxs" event={"ID":"f9e0b82d-6132-4018-9b73-cb4bb6639ba2","Type":"ContainerStarted","Data":"fa1bdcbd53452d3634cff91499fead816790f2c80ec777a8eb85e76ae812b572"}
Sep 15 18:52:11.504010 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:11.503925    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:52:11.505371 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:11.505347    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j8kt6" event={"ID":"f16c1599-2406-48bc-88fb-36e9670dc851","Type":"ContainerStarted","Data":"28e03bb68aae0903837ef23d3227350b31850410c2f661a9c9b5248a5a84fa4d"}
Sep 15 18:52:11.519508 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:11.519450    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wxhxs" podStartSLOduration=68.000152925 podStartE2EDuration="1m12.519432206s" totalImagesPullingTime="4.519279281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:52:11.519307287 +0000 UTC m=+73.006636274" watchObservedRunningTime="2026-09-15 18:52:11.519432206 +0000 UTC m=+73.006761193"
Sep 15 18:52:11.534219 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:11.534171    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j8kt6" podStartSLOduration=67.00840403 podStartE2EDuration="1m11.53415409s" totalImagesPullingTime="4.52575006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:52:11.533038026 +0000 UTC m=+73.020367013" watchObservedRunningTime="2026-09-15 18:52:11.53415409 +0000 UTC m=+73.021483075"
Sep 15 18:52:18.903256 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:18.903217    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:52:18.903862 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:18.903318    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:52:23.142972 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:23.142825    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zknv8_9c998a93-b454-41f5-a589-87f5364c062c/serve-healthcheck-canary/0.log"
Sep 15 18:52:38.908768 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:38.908737    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:52:38.912597 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:38.912575    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-57bbd7bfc6-b6nqj"
Sep 15 18:52:42.512391 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:52:42.512361    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wxhxs"
Sep 15 18:53:00.852846 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:00.852808    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:00.871726 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:00.871702    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:01.723471 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:01.723443    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:14.481174 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:14.481135    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 18:53:14.481691 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:14.481536    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="alertmanager" containerID="cri-o://08d75c28331e827e49da7e92c09d3b25336cbab9380f93741c37a1762b523c45" gracePeriod=120
Sep 15 18:53:14.481691 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:14.481626    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="kube-rbac-proxy-metric" containerID="cri-o://0ebfea9981ae1f7386f2881473362e37352c1f150b701f79c60b4732dc855e0a" gracePeriod=120
Sep 15 18:53:14.481691 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:14.481650    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="kube-rbac-proxy-web" containerID="cri-o://43db3a3ccdc557d4b83c5bb3fb51cbf0e107575eeafe9f68adc1de0f5b994d19" gracePeriod=120
Sep 15 18:53:14.481864 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:14.481681    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="kube-rbac-proxy" containerID="cri-o://35827e8da70aa5d24f59ce321828e241c27d945a78f3a92851c93665bdf3f618" gracePeriod=120
Sep 15 18:53:14.481864 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:14.481712    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="config-reloader" containerID="cri-o://c824620aa507f06b955e59c3b4e7af828c089e7b65def8ebf3004a3be22da99e" gracePeriod=120
Sep 15 18:53:14.481968 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:14.481930    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="prom-label-proxy" containerID="cri-o://b25a94eb1fbf499e7f98914b776ba5a51c7a5ef41f7ec3d7c650e468e72a3e3f" gracePeriod=120
Sep 15 18:53:14.760817 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:14.760784    2555 generic.go:356] "Generic (PLEG): container finished" podID="4411417b-070e-4696-b9ed-53d9692e16a6" containerID="b25a94eb1fbf499e7f98914b776ba5a51c7a5ef41f7ec3d7c650e468e72a3e3f" exitCode=0
Sep 15 18:53:14.760817 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:14.760810    2555 generic.go:356] "Generic (PLEG): container finished" podID="4411417b-070e-4696-b9ed-53d9692e16a6" containerID="35827e8da70aa5d24f59ce321828e241c27d945a78f3a92851c93665bdf3f618" exitCode=0
Sep 15 18:53:14.760817 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:14.760818    2555 generic.go:356] "Generic (PLEG): container finished" podID="4411417b-070e-4696-b9ed-53d9692e16a6" containerID="c824620aa507f06b955e59c3b4e7af828c089e7b65def8ebf3004a3be22da99e" exitCode=0
Sep 15 18:53:14.760817 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:14.760823    2555 generic.go:356] "Generic (PLEG): container finished" podID="4411417b-070e-4696-b9ed-53d9692e16a6" containerID="08d75c28331e827e49da7e92c09d3b25336cbab9380f93741c37a1762b523c45" exitCode=0
Sep 15 18:53:14.761102 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:14.760849    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4411417b-070e-4696-b9ed-53d9692e16a6","Type":"ContainerDied","Data":"b25a94eb1fbf499e7f98914b776ba5a51c7a5ef41f7ec3d7c650e468e72a3e3f"}
Sep 15 18:53:14.761102 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:14.760922    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4411417b-070e-4696-b9ed-53d9692e16a6","Type":"ContainerDied","Data":"35827e8da70aa5d24f59ce321828e241c27d945a78f3a92851c93665bdf3f618"}
Sep 15 18:53:14.761102 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:14.760950    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4411417b-070e-4696-b9ed-53d9692e16a6","Type":"ContainerDied","Data":"c824620aa507f06b955e59c3b4e7af828c089e7b65def8ebf3004a3be22da99e"}
Sep 15 18:53:14.761102 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:14.760973    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4411417b-070e-4696-b9ed-53d9692e16a6","Type":"ContainerDied","Data":"08d75c28331e827e49da7e92c09d3b25336cbab9380f93741c37a1762b523c45"}
Sep 15 18:53:15.721006 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.720980    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:15.769313 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.769278    2555 generic.go:356] "Generic (PLEG): container finished" podID="4411417b-070e-4696-b9ed-53d9692e16a6" containerID="0ebfea9981ae1f7386f2881473362e37352c1f150b701f79c60b4732dc855e0a" exitCode=0
Sep 15 18:53:15.769313 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.769306    2555 generic.go:356] "Generic (PLEG): container finished" podID="4411417b-070e-4696-b9ed-53d9692e16a6" containerID="43db3a3ccdc557d4b83c5bb3fb51cbf0e107575eeafe9f68adc1de0f5b994d19" exitCode=0
Sep 15 18:53:15.769513 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.769344    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4411417b-070e-4696-b9ed-53d9692e16a6","Type":"ContainerDied","Data":"0ebfea9981ae1f7386f2881473362e37352c1f150b701f79c60b4732dc855e0a"}
Sep 15 18:53:15.769513 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.769368    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:15.769513 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.769418    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4411417b-070e-4696-b9ed-53d9692e16a6","Type":"ContainerDied","Data":"43db3a3ccdc557d4b83c5bb3fb51cbf0e107575eeafe9f68adc1de0f5b994d19"}
Sep 15 18:53:15.769513 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.769462    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4411417b-070e-4696-b9ed-53d9692e16a6","Type":"ContainerDied","Data":"a931b2a413fc8623f4770bc696ee1d4ab60250d4e4356c2f5bb3764dfb1366d0"}
Sep 15 18:53:15.770071 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.770053    2555 scope.go:117] "RemoveContainer" containerID="b25a94eb1fbf499e7f98914b776ba5a51c7a5ef41f7ec3d7c650e468e72a3e3f"
Sep 15 18:53:15.778388 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.778371    2555 scope.go:117] "RemoveContainer" containerID="0ebfea9981ae1f7386f2881473362e37352c1f150b701f79c60b4732dc855e0a"
Sep 15 18:53:15.785147 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.785126    2555 scope.go:117] "RemoveContainer" containerID="35827e8da70aa5d24f59ce321828e241c27d945a78f3a92851c93665bdf3f618"
Sep 15 18:53:15.792528 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.792511    2555 scope.go:117] "RemoveContainer" containerID="43db3a3ccdc557d4b83c5bb3fb51cbf0e107575eeafe9f68adc1de0f5b994d19"
Sep 15 18:53:15.800051 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.800029    2555 scope.go:117] "RemoveContainer" containerID="c824620aa507f06b955e59c3b4e7af828c089e7b65def8ebf3004a3be22da99e"
Sep 15 18:53:15.807317 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.807284    2555 scope.go:117] "RemoveContainer" containerID="08d75c28331e827e49da7e92c09d3b25336cbab9380f93741c37a1762b523c45"
Sep 15 18:53:15.814051 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.814031    2555 scope.go:117] "RemoveContainer" containerID="881182aee9f095367413589e18c33a735f4d3322eddde72136bc835e9836c7c0"
Sep 15 18:53:15.820904 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.820889    2555 scope.go:117] "RemoveContainer" containerID="b25a94eb1fbf499e7f98914b776ba5a51c7a5ef41f7ec3d7c650e468e72a3e3f"
Sep 15 18:53:15.821182 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:53:15.821157    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25a94eb1fbf499e7f98914b776ba5a51c7a5ef41f7ec3d7c650e468e72a3e3f\": container with ID starting with b25a94eb1fbf499e7f98914b776ba5a51c7a5ef41f7ec3d7c650e468e72a3e3f not found: ID does not exist" containerID="b25a94eb1fbf499e7f98914b776ba5a51c7a5ef41f7ec3d7c650e468e72a3e3f"
Sep 15 18:53:15.821266 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.821188    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25a94eb1fbf499e7f98914b776ba5a51c7a5ef41f7ec3d7c650e468e72a3e3f"} err="failed to get container status \"b25a94eb1fbf499e7f98914b776ba5a51c7a5ef41f7ec3d7c650e468e72a3e3f\": rpc error: code = NotFound desc = could not find container \"b25a94eb1fbf499e7f98914b776ba5a51c7a5ef41f7ec3d7c650e468e72a3e3f\": container with ID starting with b25a94eb1fbf499e7f98914b776ba5a51c7a5ef41f7ec3d7c650e468e72a3e3f not found: ID does not exist"
Sep 15 18:53:15.821266 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.821221    2555 scope.go:117] "RemoveContainer" containerID="0ebfea9981ae1f7386f2881473362e37352c1f150b701f79c60b4732dc855e0a"
Sep 15 18:53:15.821459 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:53:15.821443    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ebfea9981ae1f7386f2881473362e37352c1f150b701f79c60b4732dc855e0a\": container with ID starting with 0ebfea9981ae1f7386f2881473362e37352c1f150b701f79c60b4732dc855e0a not found: ID does not exist" containerID="0ebfea9981ae1f7386f2881473362e37352c1f150b701f79c60b4732dc855e0a"
Sep 15 18:53:15.821500 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.821464    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ebfea9981ae1f7386f2881473362e37352c1f150b701f79c60b4732dc855e0a"} err="failed to get container status \"0ebfea9981ae1f7386f2881473362e37352c1f150b701f79c60b4732dc855e0a\": rpc error: code = NotFound desc = could not find container \"0ebfea9981ae1f7386f2881473362e37352c1f150b701f79c60b4732dc855e0a\": container with ID starting with 0ebfea9981ae1f7386f2881473362e37352c1f150b701f79c60b4732dc855e0a not found: ID does not exist"
Sep 15 18:53:15.821500 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.821479    2555 scope.go:117] "RemoveContainer" containerID="35827e8da70aa5d24f59ce321828e241c27d945a78f3a92851c93665bdf3f618"
Sep 15 18:53:15.821765 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:53:15.821748    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35827e8da70aa5d24f59ce321828e241c27d945a78f3a92851c93665bdf3f618\": container with ID starting with 35827e8da70aa5d24f59ce321828e241c27d945a78f3a92851c93665bdf3f618 not found: ID does not exist" containerID="35827e8da70aa5d24f59ce321828e241c27d945a78f3a92851c93665bdf3f618"
Sep 15 18:53:15.821824 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.821766    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35827e8da70aa5d24f59ce321828e241c27d945a78f3a92851c93665bdf3f618"} err="failed to get container status \"35827e8da70aa5d24f59ce321828e241c27d945a78f3a92851c93665bdf3f618\": rpc error: code = NotFound desc = could not find container \"35827e8da70aa5d24f59ce321828e241c27d945a78f3a92851c93665bdf3f618\": container with ID starting with 35827e8da70aa5d24f59ce321828e241c27d945a78f3a92851c93665bdf3f618 not found: ID does not exist"
Sep 15 18:53:15.821824 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.821784    2555 scope.go:117] "RemoveContainer" containerID="43db3a3ccdc557d4b83c5bb3fb51cbf0e107575eeafe9f68adc1de0f5b994d19"
Sep 15 18:53:15.821992 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:53:15.821978    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43db3a3ccdc557d4b83c5bb3fb51cbf0e107575eeafe9f68adc1de0f5b994d19\": container with ID starting with 43db3a3ccdc557d4b83c5bb3fb51cbf0e107575eeafe9f68adc1de0f5b994d19 not found: ID does not exist" containerID="43db3a3ccdc557d4b83c5bb3fb51cbf0e107575eeafe9f68adc1de0f5b994d19"
Sep 15 18:53:15.822033 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.821995    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43db3a3ccdc557d4b83c5bb3fb51cbf0e107575eeafe9f68adc1de0f5b994d19"} err="failed to get container status \"43db3a3ccdc557d4b83c5bb3fb51cbf0e107575eeafe9f68adc1de0f5b994d19\": rpc error: code = NotFound desc = could not find container \"43db3a3ccdc557d4b83c5bb3fb51cbf0e107575eeafe9f68adc1de0f5b994d19\": container with ID starting with 43db3a3ccdc557d4b83c5bb3fb51cbf0e107575eeafe9f68adc1de0f5b994d19 not found: ID does not exist"
Sep 15 18:53:15.822033 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.822016    2555 scope.go:117] "RemoveContainer" containerID="c824620aa507f06b955e59c3b4e7af828c089e7b65def8ebf3004a3be22da99e"
Sep 15 18:53:15.822209 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:53:15.822194    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c824620aa507f06b955e59c3b4e7af828c089e7b65def8ebf3004a3be22da99e\": container with ID starting with c824620aa507f06b955e59c3b4e7af828c089e7b65def8ebf3004a3be22da99e not found: ID does not exist" containerID="c824620aa507f06b955e59c3b4e7af828c089e7b65def8ebf3004a3be22da99e"
Sep 15 18:53:15.822249 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.822210    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c824620aa507f06b955e59c3b4e7af828c089e7b65def8ebf3004a3be22da99e"} err="failed to get container status \"c824620aa507f06b955e59c3b4e7af828c089e7b65def8ebf3004a3be22da99e\": rpc error: code = NotFound desc = could not find container \"c824620aa507f06b955e59c3b4e7af828c089e7b65def8ebf3004a3be22da99e\": container with ID starting with c824620aa507f06b955e59c3b4e7af828c089e7b65def8ebf3004a3be22da99e not found: ID does not exist"
Sep 15 18:53:15.822249 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.822221    2555 scope.go:117] "RemoveContainer" containerID="08d75c28331e827e49da7e92c09d3b25336cbab9380f93741c37a1762b523c45"
Sep 15 18:53:15.822421 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:53:15.822400    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d75c28331e827e49da7e92c09d3b25336cbab9380f93741c37a1762b523c45\": container with ID starting with 08d75c28331e827e49da7e92c09d3b25336cbab9380f93741c37a1762b523c45 not found: ID does not exist" containerID="08d75c28331e827e49da7e92c09d3b25336cbab9380f93741c37a1762b523c45"
Sep 15 18:53:15.822457 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.822417    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d75c28331e827e49da7e92c09d3b25336cbab9380f93741c37a1762b523c45"} err="failed to get container status \"08d75c28331e827e49da7e92c09d3b25336cbab9380f93741c37a1762b523c45\": rpc error: code = NotFound desc = could not find container \"08d75c28331e827e49da7e92c09d3b25336cbab9380f93741c37a1762b523c45\": container with ID starting with 08d75c28331e827e49da7e92c09d3b25336cbab9380f93741c37a1762b523c45 not found: ID does not exist"
Sep 15 18:53:15.822457 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.822431    2555 scope.go:117] "RemoveContainer" containerID="881182aee9f095367413589e18c33a735f4d3322eddde72136bc835e9836c7c0"
Sep 15 18:53:15.822676 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:53:15.822658    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"881182aee9f095367413589e18c33a735f4d3322eddde72136bc835e9836c7c0\": container with ID starting with 881182aee9f095367413589e18c33a735f4d3322eddde72136bc835e9836c7c0 not found: ID does not exist" containerID="881182aee9f095367413589e18c33a735f4d3322eddde72136bc835e9836c7c0"
Sep 15 18:53:15.822742 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.822679    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"881182aee9f095367413589e18c33a735f4d3322eddde72136bc835e9836c7c0"} err="failed to get container status \"881182aee9f095367413589e18c33a735f4d3322eddde72136bc835e9836c7c0\": rpc error: code = NotFound desc = could not find container \"881182aee9f095367413589e18c33a735f4d3322eddde72136bc835e9836c7c0\": container with ID starting with 881182aee9f095367413589e18c33a735f4d3322eddde72136bc835e9836c7c0 not found: ID does not exist"
Sep 15 18:53:15.822742 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.822690    2555 scope.go:117] "RemoveContainer" containerID="b25a94eb1fbf499e7f98914b776ba5a51c7a5ef41f7ec3d7c650e468e72a3e3f"
Sep 15 18:53:15.822895 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.822880    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25a94eb1fbf499e7f98914b776ba5a51c7a5ef41f7ec3d7c650e468e72a3e3f"} err="failed to get container status \"b25a94eb1fbf499e7f98914b776ba5a51c7a5ef41f7ec3d7c650e468e72a3e3f\": rpc error: code = NotFound desc = could not find container \"b25a94eb1fbf499e7f98914b776ba5a51c7a5ef41f7ec3d7c650e468e72a3e3f\": container with ID starting with b25a94eb1fbf499e7f98914b776ba5a51c7a5ef41f7ec3d7c650e468e72a3e3f not found: ID does not exist"
Sep 15 18:53:15.822942 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.822895    2555 scope.go:117] "RemoveContainer" containerID="0ebfea9981ae1f7386f2881473362e37352c1f150b701f79c60b4732dc855e0a"
Sep 15 18:53:15.823075 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.823059    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ebfea9981ae1f7386f2881473362e37352c1f150b701f79c60b4732dc855e0a"} err="failed to get container status \"0ebfea9981ae1f7386f2881473362e37352c1f150b701f79c60b4732dc855e0a\": rpc error: code = NotFound desc = could not find container \"0ebfea9981ae1f7386f2881473362e37352c1f150b701f79c60b4732dc855e0a\": container with ID starting with 0ebfea9981ae1f7386f2881473362e37352c1f150b701f79c60b4732dc855e0a not found: ID does not exist"
Sep 15 18:53:15.823118 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.823076    2555 scope.go:117] "RemoveContainer" containerID="35827e8da70aa5d24f59ce321828e241c27d945a78f3a92851c93665bdf3f618"
Sep 15 18:53:15.823299 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.823282    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35827e8da70aa5d24f59ce321828e241c27d945a78f3a92851c93665bdf3f618"} err="failed to get container status \"35827e8da70aa5d24f59ce321828e241c27d945a78f3a92851c93665bdf3f618\": rpc error: code = NotFound desc = could not find container \"35827e8da70aa5d24f59ce321828e241c27d945a78f3a92851c93665bdf3f618\": container with ID starting with 35827e8da70aa5d24f59ce321828e241c27d945a78f3a92851c93665bdf3f618 not found: ID does not exist"
Sep 15 18:53:15.823299 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.823299    2555 scope.go:117] "RemoveContainer" containerID="43db3a3ccdc557d4b83c5bb3fb51cbf0e107575eeafe9f68adc1de0f5b994d19"
Sep 15 18:53:15.823492 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.823475    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43db3a3ccdc557d4b83c5bb3fb51cbf0e107575eeafe9f68adc1de0f5b994d19"} err="failed to get container status \"43db3a3ccdc557d4b83c5bb3fb51cbf0e107575eeafe9f68adc1de0f5b994d19\": rpc error: code = NotFound desc = could not find container \"43db3a3ccdc557d4b83c5bb3fb51cbf0e107575eeafe9f68adc1de0f5b994d19\": container with ID starting with 43db3a3ccdc557d4b83c5bb3fb51cbf0e107575eeafe9f68adc1de0f5b994d19 not found: ID does not exist"
Sep 15 18:53:15.823534 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.823491    2555 scope.go:117] "RemoveContainer" containerID="c824620aa507f06b955e59c3b4e7af828c089e7b65def8ebf3004a3be22da99e"
Sep 15 18:53:15.823765 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.823740    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c824620aa507f06b955e59c3b4e7af828c089e7b65def8ebf3004a3be22da99e"} err="failed to get container status \"c824620aa507f06b955e59c3b4e7af828c089e7b65def8ebf3004a3be22da99e\": rpc error: code = NotFound desc = could not find container \"c824620aa507f06b955e59c3b4e7af828c089e7b65def8ebf3004a3be22da99e\": container with ID starting with c824620aa507f06b955e59c3b4e7af828c089e7b65def8ebf3004a3be22da99e not found: ID does not exist"
Sep 15 18:53:15.823765 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.823756    2555 scope.go:117] "RemoveContainer" containerID="08d75c28331e827e49da7e92c09d3b25336cbab9380f93741c37a1762b523c45"
Sep 15 18:53:15.823997 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.823978    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d75c28331e827e49da7e92c09d3b25336cbab9380f93741c37a1762b523c45"} err="failed to get container status \"08d75c28331e827e49da7e92c09d3b25336cbab9380f93741c37a1762b523c45\": rpc error: code = NotFound desc = could not find container \"08d75c28331e827e49da7e92c09d3b25336cbab9380f93741c37a1762b523c45\": container with ID starting with 08d75c28331e827e49da7e92c09d3b25336cbab9380f93741c37a1762b523c45 not found: ID does not exist"
Sep 15 18:53:15.824045 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.823997    2555 scope.go:117] "RemoveContainer" containerID="881182aee9f095367413589e18c33a735f4d3322eddde72136bc835e9836c7c0"
Sep 15 18:53:15.824228 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.824212    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"881182aee9f095367413589e18c33a735f4d3322eddde72136bc835e9836c7c0"} err="failed to get container status \"881182aee9f095367413589e18c33a735f4d3322eddde72136bc835e9836c7c0\": rpc error: code = NotFound desc = could not find container \"881182aee9f095367413589e18c33a735f4d3322eddde72136bc835e9836c7c0\": container with ID starting with 881182aee9f095367413589e18c33a735f4d3322eddde72136bc835e9836c7c0 not found: ID does not exist"
Sep 15 18:53:15.875746 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.875723    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-cluster-tls-config\") pod \"4411417b-070e-4696-b9ed-53d9692e16a6\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") "
Sep 15 18:53:15.875879 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.875757    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-main-tls\") pod \"4411417b-070e-4696-b9ed-53d9692e16a6\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") "
Sep 15 18:53:15.875879 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.875783    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4411417b-070e-4696-b9ed-53d9692e16a6-metrics-client-ca\") pod \"4411417b-070e-4696-b9ed-53d9692e16a6\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") "
Sep 15 18:53:15.875879 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.875804    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-kube-rbac-proxy-web\") pod \"4411417b-070e-4696-b9ed-53d9692e16a6\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") "
Sep 15 18:53:15.875879 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.875848    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-config-volume\") pod \"4411417b-070e-4696-b9ed-53d9692e16a6\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") "
Sep 15 18:53:15.875879 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.875871    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4411417b-070e-4696-b9ed-53d9692e16a6-alertmanager-main-db\") pod \"4411417b-070e-4696-b9ed-53d9692e16a6\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") "
Sep 15 18:53:15.876126 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.875921    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"4411417b-070e-4696-b9ed-53d9692e16a6\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") "
Sep 15 18:53:15.876126 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.875975    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-web-config\") pod \"4411417b-070e-4696-b9ed-53d9692e16a6\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") "
Sep 15 18:53:15.876126 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.876024    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4411417b-070e-4696-b9ed-53d9692e16a6-alertmanager-trusted-ca-bundle\") pod \"4411417b-070e-4696-b9ed-53d9692e16a6\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") "
Sep 15 18:53:15.876126 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.876089    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4411417b-070e-4696-b9ed-53d9692e16a6-config-out\") pod \"4411417b-070e-4696-b9ed-53d9692e16a6\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") "
Sep 15 18:53:15.876126 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.876118    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4411417b-070e-4696-b9ed-53d9692e16a6-tls-assets\") pod \"4411417b-070e-4696-b9ed-53d9692e16a6\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") "
Sep 15 18:53:15.876374 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.876151    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5tlx\" (UniqueName: \"kubernetes.io/projected/4411417b-070e-4696-b9ed-53d9692e16a6-kube-api-access-m5tlx\") pod \"4411417b-070e-4696-b9ed-53d9692e16a6\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") "
Sep 15 18:53:15.876374 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.876159    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4411417b-070e-4696-b9ed-53d9692e16a6-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "4411417b-070e-4696-b9ed-53d9692e16a6" (UID: "4411417b-070e-4696-b9ed-53d9692e16a6"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:53:15.876374 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.876175    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-kube-rbac-proxy\") pod \"4411417b-070e-4696-b9ed-53d9692e16a6\" (UID: \"4411417b-070e-4696-b9ed-53d9692e16a6\") "
Sep 15 18:53:15.876374 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.876219    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4411417b-070e-4696-b9ed-53d9692e16a6-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "4411417b-070e-4696-b9ed-53d9692e16a6" (UID: "4411417b-070e-4696-b9ed-53d9692e16a6"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 18:53:15.876573 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.876452    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4411417b-070e-4696-b9ed-53d9692e16a6-metrics-client-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:15.876573 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.876473    2555 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4411417b-070e-4696-b9ed-53d9692e16a6-alertmanager-main-db\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:15.877203 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.877158    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4411417b-070e-4696-b9ed-53d9692e16a6-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "4411417b-070e-4696-b9ed-53d9692e16a6" (UID: "4411417b-070e-4696-b9ed-53d9692e16a6"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:53:15.878770 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.878741    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-config-volume" (OuterVolumeSpecName: "config-volume") pod "4411417b-070e-4696-b9ed-53d9692e16a6" (UID: "4411417b-070e-4696-b9ed-53d9692e16a6"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:53:15.879181 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.879151    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4411417b-070e-4696-b9ed-53d9692e16a6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4411417b-070e-4696-b9ed-53d9692e16a6" (UID: "4411417b-070e-4696-b9ed-53d9692e16a6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:53:15.879540 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.879495    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "4411417b-070e-4696-b9ed-53d9692e16a6" (UID: "4411417b-070e-4696-b9ed-53d9692e16a6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:53:15.879657 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.879593    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4411417b-070e-4696-b9ed-53d9692e16a6-kube-api-access-m5tlx" (OuterVolumeSpecName: "kube-api-access-m5tlx") pod "4411417b-070e-4696-b9ed-53d9692e16a6" (UID: "4411417b-070e-4696-b9ed-53d9692e16a6"). InnerVolumeSpecName "kube-api-access-m5tlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:53:15.879727 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.879657    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4411417b-070e-4696-b9ed-53d9692e16a6-config-out" (OuterVolumeSpecName: "config-out") pod "4411417b-070e-4696-b9ed-53d9692e16a6" (UID: "4411417b-070e-4696-b9ed-53d9692e16a6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 18:53:15.880097 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.880074    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "4411417b-070e-4696-b9ed-53d9692e16a6" (UID: "4411417b-070e-4696-b9ed-53d9692e16a6"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:53:15.880205 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.880146    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "4411417b-070e-4696-b9ed-53d9692e16a6" (UID: "4411417b-070e-4696-b9ed-53d9692e16a6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:53:15.880292 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.880277    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "4411417b-070e-4696-b9ed-53d9692e16a6" (UID: "4411417b-070e-4696-b9ed-53d9692e16a6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:53:15.884632 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.884584    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "4411417b-070e-4696-b9ed-53d9692e16a6" (UID: "4411417b-070e-4696-b9ed-53d9692e16a6"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:53:15.890784 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.890763    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-web-config" (OuterVolumeSpecName: "web-config") pod "4411417b-070e-4696-b9ed-53d9692e16a6" (UID: "4411417b-070e-4696-b9ed-53d9692e16a6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:53:15.976951 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.976914    2555 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-cluster-tls-config\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:15.976951 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.976947    2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-main-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:15.976951 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.976960    2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:15.977195 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.976970    2555 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-config-volume\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:15.977195 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.976979    2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:15.977195 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.976989    2555 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-web-config\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:15.977195 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.976997    2555 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4411417b-070e-4696-b9ed-53d9692e16a6-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:15.977195 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.977005    2555 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4411417b-070e-4696-b9ed-53d9692e16a6-config-out\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:15.977195 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.977013    2555 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4411417b-070e-4696-b9ed-53d9692e16a6-tls-assets\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:15.977195 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.977021    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5tlx\" (UniqueName: \"kubernetes.io/projected/4411417b-070e-4696-b9ed-53d9692e16a6-kube-api-access-m5tlx\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:15.977195 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:15.977029    2555 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4411417b-070e-4696-b9ed-53d9692e16a6-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:16.092249 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.092218    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 18:53:16.095732 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.095706    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 18:53:16.120322 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120295    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 18:53:16.120591 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120579    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="prom-label-proxy"
Sep 15 18:53:16.120648 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120592    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="prom-label-proxy"
Sep 15 18:53:16.120648 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120602    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="kube-rbac-proxy"
Sep 15 18:53:16.120648 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120620    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="kube-rbac-proxy"
Sep 15 18:53:16.120648 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120628    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="init-config-reloader"
Sep 15 18:53:16.120648 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120633    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="init-config-reloader"
Sep 15 18:53:16.120648 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120642    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="config-reloader"
Sep 15 18:53:16.120648 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120647    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="config-reloader"
Sep 15 18:53:16.120884 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120657    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="kube-rbac-proxy-web"
Sep 15 18:53:16.120884 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120663    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="kube-rbac-proxy-web"
Sep 15 18:53:16.120884 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120669    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="kube-rbac-proxy-metric"
Sep 15 18:53:16.120884 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120675    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="kube-rbac-proxy-metric"
Sep 15 18:53:16.120884 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120690    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="alertmanager"
Sep 15 18:53:16.120884 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120694    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="alertmanager"
Sep 15 18:53:16.120884 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120748    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="prom-label-proxy"
Sep 15 18:53:16.120884 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120758    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="kube-rbac-proxy-metric"
Sep 15 18:53:16.120884 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120764    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="kube-rbac-proxy-web"
Sep 15 18:53:16.120884 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120772    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="kube-rbac-proxy"
Sep 15 18:53:16.120884 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120777    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="alertmanager"
Sep 15 18:53:16.120884 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.120783    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" containerName="config-reloader"
Sep 15 18:53:16.128532 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.128507    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.130826 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.130800    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 15 18:53:16.130946 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.130801    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 15 18:53:16.131022 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.131003    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 15 18:53:16.131080 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.131022    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 15 18:53:16.131080 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.131039    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 15 18:53:16.131178 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.131132    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 15 18:53:16.131178 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.131142    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 15 18:53:16.131178 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.131169    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 15 18:53:16.131348 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.131334    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-j9kfz\""
Sep 15 18:53:16.135173 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.135152    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 15 18:53:16.139008 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.138986    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 18:53:16.278767 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.278739    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.278767 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.278773    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.279007 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.278793    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.279007 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.278808    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.279007 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.278828    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-config-volume\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.279007 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.278845    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.279007 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.278923    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.279007 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.278966    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-config-out\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.279234 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.279034    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.279234 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.279076    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-web-config\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.279234 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.279103    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.279234 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.279189    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.279234 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.279212    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69m2w\" (UniqueName: \"kubernetes.io/projected/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-kube-api-access-69m2w\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.379683 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.379566    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.379683 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.379601    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.379914 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.379724    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-config-out\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.379914 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.379758    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.379914 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.379777    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-web-config\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.379914 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.379794    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.379914 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.379837    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.380157 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.379949    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69m2w\" (UniqueName: \"kubernetes.io/projected/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-kube-api-access-69m2w\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.380157 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.379989    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.380157 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.380023    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.380157 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.380037    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.380157 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.380052    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.380157 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.380077    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.380157 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.380111    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-config-volume\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.381360 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.381296    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.381989 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.381966    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.382914 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.382890    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.383078 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.383029    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-config-out\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.383186 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.383154    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.383305 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.383201    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-config-volume\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.383305 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.383256    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.383410 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.383305    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.383410 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.383325    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.384278 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.384259    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-web-config\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.384364 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.384351    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.387177 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.387162    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69m2w\" (UniqueName: \"kubernetes.io/projected/06cd018b-7df9-4f47-a5b2-c8f9d18aa94e-kube-api-access-69m2w\") pod \"alertmanager-main-0\" (UID: \"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.438741 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.438710    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 18:53:16.715829 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.715806    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 18:53:16.717653 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:53:16.717622    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06cd018b_7df9_4f47_a5b2_c8f9d18aa94e.slice/crio-24999f26035cb28debf3c79166aa26340004e1f18be46ca77135370800f582dd WatchSource:0}: Error finding container 24999f26035cb28debf3c79166aa26340004e1f18be46ca77135370800f582dd: Status 404 returned error can't find the container with id 24999f26035cb28debf3c79166aa26340004e1f18be46ca77135370800f582dd
Sep 15 18:53:16.774912 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:16.774875    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e","Type":"ContainerStarted","Data":"24999f26035cb28debf3c79166aa26340004e1f18be46ca77135370800f582dd"}
Sep 15 18:53:17.038557 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:17.038524    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4411417b-070e-4696-b9ed-53d9692e16a6" path="/var/lib/kubelet/pods/4411417b-070e-4696-b9ed-53d9692e16a6/volumes"
Sep 15 18:53:17.781662 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:17.781629    2555 generic.go:356] "Generic (PLEG): container finished" podID="06cd018b-7df9-4f47-a5b2-c8f9d18aa94e" containerID="3c89457dce3c75224e32f7d023ed0df713480f13924871c29dc6011d7620deb3" exitCode=0
Sep 15 18:53:17.782041 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:17.781669    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e","Type":"ContainerDied","Data":"3c89457dce3c75224e32f7d023ed0df713480f13924871c29dc6011d7620deb3"}
Sep 15 18:53:18.535404 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.535364    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-8545d8745b-fkp52"]
Sep 15 18:53:18.538388 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.538364    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.540658 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.540635    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 15 18:53:18.540787 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.540637    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 15 18:53:18.540787 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.540688    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 15 18:53:18.540787 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.540700    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7vpxb\""
Sep 15 18:53:18.540929 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.540877    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 15 18:53:18.540929 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.540918    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 15 18:53:18.546803 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.546774    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 15 18:53:18.549898 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.549873    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8545d8745b-fkp52"]
Sep 15 18:53:18.700835 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.700795    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/695b920f-2498-4cc6-8d0e-1095adfb7f2a-secret-telemeter-client\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.700835 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.700846    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/695b920f-2498-4cc6-8d0e-1095adfb7f2a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.701039 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.700883    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/695b920f-2498-4cc6-8d0e-1095adfb7f2a-serving-certs-ca-bundle\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.701039 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.700900    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/695b920f-2498-4cc6-8d0e-1095adfb7f2a-telemeter-client-tls\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.701039 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.700916    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/695b920f-2498-4cc6-8d0e-1095adfb7f2a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.701039 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.700934    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/695b920f-2498-4cc6-8d0e-1095adfb7f2a-metrics-client-ca\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.701039 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.700952    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gt2r\" (UniqueName: \"kubernetes.io/projected/695b920f-2498-4cc6-8d0e-1095adfb7f2a-kube-api-access-8gt2r\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.701039 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.700975    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/695b920f-2498-4cc6-8d0e-1095adfb7f2a-federate-client-tls\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.790562 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.790476    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e","Type":"ContainerStarted","Data":"8c909929d5677b813a085990ec559be48e1ee4fb18125602c54cfb490f7317f2"}
Sep 15 18:53:18.790964 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.790562    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e","Type":"ContainerStarted","Data":"ae1e9a8f0551c9d6fc95ff462b6e17528a298fcc924f1ed4853586113414d423"}
Sep 15 18:53:18.790964 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.790635    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e","Type":"ContainerStarted","Data":"9f7d6c2e76dec358592d1467c1fe79f5570b26bc111ff8bc474f9b99e0c5d29d"}
Sep 15 18:53:18.790964 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.790678    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e","Type":"ContainerStarted","Data":"8b532313484ab5edad523f0632533755e73290afc0411b1d39764c8e2f9851c2"}
Sep 15 18:53:18.790964 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.790720    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e","Type":"ContainerStarted","Data":"50706f9ae681ee1444d23a25f4eea0228c5ab938622f345c17707fcbab0cea79"}
Sep 15 18:53:18.790964 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.790758    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06cd018b-7df9-4f47-a5b2-c8f9d18aa94e","Type":"ContainerStarted","Data":"8484ba74f5ef11b265b1882f8ccd8f29639e3b579a553c731df8efa99e99206e"}
Sep 15 18:53:18.801391 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.801363    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/695b920f-2498-4cc6-8d0e-1095adfb7f2a-secret-telemeter-client\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.801571 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.801397    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/695b920f-2498-4cc6-8d0e-1095adfb7f2a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.801571 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.801424    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/695b920f-2498-4cc6-8d0e-1095adfb7f2a-serving-certs-ca-bundle\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.801571 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.801440    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/695b920f-2498-4cc6-8d0e-1095adfb7f2a-telemeter-client-tls\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.801571 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.801461    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/695b920f-2498-4cc6-8d0e-1095adfb7f2a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.801571 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.801488    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/695b920f-2498-4cc6-8d0e-1095adfb7f2a-metrics-client-ca\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.802477 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.802011    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gt2r\" (UniqueName: \"kubernetes.io/projected/695b920f-2498-4cc6-8d0e-1095adfb7f2a-kube-api-access-8gt2r\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.802477 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.802073    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/695b920f-2498-4cc6-8d0e-1095adfb7f2a-federate-client-tls\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.802477 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.802237    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/695b920f-2498-4cc6-8d0e-1095adfb7f2a-serving-certs-ca-bundle\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.802477 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.802271    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/695b920f-2498-4cc6-8d0e-1095adfb7f2a-metrics-client-ca\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.802477 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.802396    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/695b920f-2498-4cc6-8d0e-1095adfb7f2a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.807035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.804511    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/695b920f-2498-4cc6-8d0e-1095adfb7f2a-telemeter-client-tls\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.807035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.805009    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/695b920f-2498-4cc6-8d0e-1095adfb7f2a-federate-client-tls\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.807035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.805434    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/695b920f-2498-4cc6-8d0e-1095adfb7f2a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.807035 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.805978    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/695b920f-2498-4cc6-8d0e-1095adfb7f2a-secret-telemeter-client\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.809290 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.809257    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 18:53:18.809663 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.809636    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="prometheus" containerID="cri-o://f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2" gracePeriod=600
Sep 15 18:53:18.809762 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.809651    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="kube-rbac-proxy" containerID="cri-o://8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9" gracePeriod=600
Sep 15 18:53:18.809762 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.809670    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="kube-rbac-proxy-thanos" containerID="cri-o://64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe" gracePeriod=600
Sep 15 18:53:18.809762 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.809678    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="thanos-sidecar" containerID="cri-o://aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6" gracePeriod=600
Sep 15 18:53:18.809918 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.809764    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="kube-rbac-proxy-web" containerID="cri-o://801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb" gracePeriod=600
Sep 15 18:53:18.809918 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.809798    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="config-reloader" containerID="cri-o://35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda" gracePeriod=600
Sep 15 18:53:18.811604 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.811448    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gt2r\" (UniqueName: \"kubernetes.io/projected/695b920f-2498-4cc6-8d0e-1095adfb7f2a-kube-api-access-8gt2r\") pod \"telemeter-client-8545d8745b-fkp52\" (UID: \"695b920f-2498-4cc6-8d0e-1095adfb7f2a\") " pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:18.822514 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.821865    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.806728419 podStartE2EDuration="2.821844131s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.115712ms" isStatefulPod=false podCreationTimestamp="2026-09-15 18:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:53:18.82038849 +0000 UTC m=+140.307717476" watchObservedRunningTime="2026-09-15 18:53:18.821844131 +0000 UTC m=+140.309173116"
Sep 15 18:53:18.860452 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:18.860426    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52"
Sep 15 18:53:19.083229 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.083201    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.150638 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.147864    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8545d8745b-fkp52"]
Sep 15 18:53:19.150638 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:53:19.148037    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695b920f_2498_4cc6_8d0e_1095adfb7f2a.slice/crio-6a30d254a9395d4ca3cadc9ae81808743eccd53445b0ebdfffe502e326fa2d46 WatchSource:0}: Error finding container 6a30d254a9395d4ca3cadc9ae81808743eccd53445b0ebdfffe502e326fa2d46: Status 404 returned error can't find the container with id 6a30d254a9395d4ca3cadc9ae81808743eccd53445b0ebdfffe502e326fa2d46
Sep 15 18:53:19.206243 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206210    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-configmap-kubelet-serving-ca-bundle\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.206404 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206251    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-rulefiles-2\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.206404 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206276    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-thanos-prometheus-http-client-file\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.206404 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206317    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-configmap-metrics-client-ca\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.206404 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206343    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-kube-rbac-proxy\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.206404 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206381    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-tls-assets\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.206697 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206412    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-configmap-serving-certs-ca-bundle\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.206697 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206452    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.206697 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206476    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-config-out\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.206697 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206498    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-config\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.206697 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206521    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-metrics-client-certs\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.206697 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206555    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-grpc-tls\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.206697 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206581    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-trusted-ca-bundle\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.206697 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206646    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlgpq\" (UniqueName: \"kubernetes.io/projected/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-kube-api-access-zlgpq\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.206697 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206652    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:53:19.206697 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206682    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-rulefiles-0\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.207195 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206714    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.207195 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206737    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:53:19.207195 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206748    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:53:19.207195 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.206760    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-rulefiles-1\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.207386 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.207284    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:53:19.207885 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.207857    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:53:19.208129 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.207969    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-db\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.208281 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.208263    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-prometheus-k8s-tls\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.208420 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.208403    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-web-config\") pod \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\" (UID: \"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41\") "
Sep 15 18:53:19.208789 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.208770    2555 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.208923 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.208906    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.209004 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.208994    2555 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-configmap-metrics-client-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.209079 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.209068    2555 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.209167 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.209155    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.209251 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.208110    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:53:19.209334 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.209069    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:53:19.209412 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.209125    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 18:53:19.209905 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.209830    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:53:19.210186 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.210139    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:53:19.210524 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.210468    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:53:19.210715 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.210682    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:53:19.210802 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.210733    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-kube-api-access-zlgpq" (OuterVolumeSpecName: "kube-api-access-zlgpq") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "kube-api-access-zlgpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:53:19.210802 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.210739    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-config" (OuterVolumeSpecName: "config") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:53:19.210802 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.210784    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:53:19.211026 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.210930    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-config-out" (OuterVolumeSpecName: "config-out") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 18:53:19.211355 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.211325    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:53:19.211767 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.211742    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:53:19.212034 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.212011    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:53:19.221420 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.221397    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-web-config" (OuterVolumeSpecName: "web-config") pod "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" (UID: "97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:53:19.309583 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.309549    2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.309583 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.309580    2555 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-config-out\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.309583 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.309590    2555 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-config\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.309836 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.309599    2555 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-metrics-client-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.309836 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.309619    2555 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-grpc-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.309836 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.309628    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlgpq\" (UniqueName: \"kubernetes.io/projected/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-kube-api-access-zlgpq\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.309836 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.309637    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.309836 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.309646    2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.309836 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.309654    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.309836 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.309662    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-prometheus-k8s-db\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.309836 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.309670    2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.309836 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.309679    2555 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-web-config\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.309836 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.309687    2555 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.309836 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.309695    2555 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-secret-kube-rbac-proxy\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.309836 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.309703    2555 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41-tls-assets\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:53:19.795837 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.795794    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52" event={"ID":"695b920f-2498-4cc6-8d0e-1095adfb7f2a","Type":"ContainerStarted","Data":"6a30d254a9395d4ca3cadc9ae81808743eccd53445b0ebdfffe502e326fa2d46"}
Sep 15 18:53:19.798980 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.798953    2555 generic.go:356] "Generic (PLEG): container finished" podID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerID="64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe" exitCode=0
Sep 15 18:53:19.798980 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.798976    2555 generic.go:356] "Generic (PLEG): container finished" podID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerID="8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9" exitCode=0
Sep 15 18:53:19.798980 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.798986    2555 generic.go:356] "Generic (PLEG): container finished" podID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerID="801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb" exitCode=0
Sep 15 18:53:19.799199 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.798994    2555 generic.go:356] "Generic (PLEG): container finished" podID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerID="aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6" exitCode=0
Sep 15 18:53:19.799199 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.799001    2555 generic.go:356] "Generic (PLEG): container finished" podID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerID="35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda" exitCode=0
Sep 15 18:53:19.799199 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.799008    2555 generic.go:356] "Generic (PLEG): container finished" podID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerID="f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2" exitCode=0
Sep 15 18:53:19.799199 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.799034    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41","Type":"ContainerDied","Data":"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe"}
Sep 15 18:53:19.799199 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.799094    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41","Type":"ContainerDied","Data":"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9"}
Sep 15 18:53:19.799199 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.799043    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.799199 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.799131    2555 scope.go:117] "RemoveContainer" containerID="64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe"
Sep 15 18:53:19.799199 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.799121    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41","Type":"ContainerDied","Data":"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb"}
Sep 15 18:53:19.799484 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.799249    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41","Type":"ContainerDied","Data":"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6"}
Sep 15 18:53:19.799484 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.799287    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41","Type":"ContainerDied","Data":"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda"}
Sep 15 18:53:19.799484 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.799327    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41","Type":"ContainerDied","Data":"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2"}
Sep 15 18:53:19.799484 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.799361    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41","Type":"ContainerDied","Data":"3f907ca59b197e5a5148f98b53526e0c5d7ff6aab80e037246763a15835ad072"}
Sep 15 18:53:19.807156 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.807139    2555 scope.go:117] "RemoveContainer" containerID="8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9"
Sep 15 18:53:19.814575 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.814434    2555 scope.go:117] "RemoveContainer" containerID="801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb"
Sep 15 18:53:19.820785 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.820763    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 18:53:19.821587 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.821573    2555 scope.go:117] "RemoveContainer" containerID="aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6"
Sep 15 18:53:19.826169 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.826147    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 18:53:19.828867 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.828852    2555 scope.go:117] "RemoveContainer" containerID="35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda"
Sep 15 18:53:19.835470 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.835452    2555 scope.go:117] "RemoveContainer" containerID="f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2"
Sep 15 18:53:19.843010 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.842992    2555 scope.go:117] "RemoveContainer" containerID="6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406"
Sep 15 18:53:19.849711 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.849693    2555 scope.go:117] "RemoveContainer" containerID="64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe"
Sep 15 18:53:19.850081 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:53:19.850001    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe\": container with ID starting with 64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe not found: ID does not exist" containerID="64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe"
Sep 15 18:53:19.850081 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.850034    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe"} err="failed to get container status \"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe\": rpc error: code = NotFound desc = could not find container \"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe\": container with ID starting with 64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe not found: ID does not exist"
Sep 15 18:53:19.850081 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.850056    2555 scope.go:117] "RemoveContainer" containerID="8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9"
Sep 15 18:53:19.850397 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:53:19.850366    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9\": container with ID starting with 8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9 not found: ID does not exist" containerID="8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9"
Sep 15 18:53:19.850505 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.850405    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9"} err="failed to get container status \"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9\": rpc error: code = NotFound desc = could not find container \"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9\": container with ID starting with 8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9 not found: ID does not exist"
Sep 15 18:53:19.850505 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.850427    2555 scope.go:117] "RemoveContainer" containerID="801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb"
Sep 15 18:53:19.850780 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:53:19.850758    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb\": container with ID starting with 801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb not found: ID does not exist" containerID="801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb"
Sep 15 18:53:19.850866 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.850783    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb"} err="failed to get container status \"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb\": rpc error: code = NotFound desc = could not find container \"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb\": container with ID starting with 801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb not found: ID does not exist"
Sep 15 18:53:19.850866 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.850808    2555 scope.go:117] "RemoveContainer" containerID="aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6"
Sep 15 18:53:19.851066 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:53:19.851046    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6\": container with ID starting with aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6 not found: ID does not exist" containerID="aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6"
Sep 15 18:53:19.851126 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851074    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6"} err="failed to get container status \"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6\": rpc error: code = NotFound desc = could not find container \"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6\": container with ID starting with aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6 not found: ID does not exist"
Sep 15 18:53:19.851126 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851098    2555 scope.go:117] "RemoveContainer" containerID="35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda"
Sep 15 18:53:19.851353 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851333    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 18:53:19.851420 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:53:19.851360    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda\": container with ID starting with 35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda not found: ID does not exist" containerID="35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda"
Sep 15 18:53:19.851420 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851375    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda"} err="failed to get container status \"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda\": rpc error: code = NotFound desc = could not find container \"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda\": container with ID starting with 35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda not found: ID does not exist"
Sep 15 18:53:19.851420 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851387    2555 scope.go:117] "RemoveContainer" containerID="f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2"
Sep 15 18:53:19.851682 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:53:19.851666    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2\": container with ID starting with f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2 not found: ID does not exist" containerID="f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2"
Sep 15 18:53:19.851735 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851684    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2"} err="failed to get container status \"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2\": rpc error: code = NotFound desc = could not find container \"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2\": container with ID starting with f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2 not found: ID does not exist"
Sep 15 18:53:19.851735 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851696    2555 scope.go:117] "RemoveContainer" containerID="6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406"
Sep 15 18:53:19.851735 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851702    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="init-config-reloader"
Sep 15 18:53:19.851735 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851718    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="init-config-reloader"
Sep 15 18:53:19.851735 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851735    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="config-reloader"
Sep 15 18:53:19.851917 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851741    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="config-reloader"
Sep 15 18:53:19.851917 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851748    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="thanos-sidecar"
Sep 15 18:53:19.851917 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851754    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="thanos-sidecar"
Sep 15 18:53:19.851917 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851760    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="kube-rbac-proxy"
Sep 15 18:53:19.851917 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851767    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="kube-rbac-proxy"
Sep 15 18:53:19.851917 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851776    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="kube-rbac-proxy-web"
Sep 15 18:53:19.851917 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851785    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="kube-rbac-proxy-web"
Sep 15 18:53:19.851917 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851797    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="prometheus"
Sep 15 18:53:19.851917 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851802    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="prometheus"
Sep 15 18:53:19.851917 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851808    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="kube-rbac-proxy-thanos"
Sep 15 18:53:19.851917 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851813    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="kube-rbac-proxy-thanos"
Sep 15 18:53:19.851917 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851860    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="config-reloader"
Sep 15 18:53:19.851917 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851872    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="kube-rbac-proxy-thanos"
Sep 15 18:53:19.851917 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851879    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="kube-rbac-proxy-web"
Sep 15 18:53:19.851917 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851884    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="thanos-sidecar"
Sep 15 18:53:19.851917 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851889    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="prometheus"
Sep 15 18:53:19.851917 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851896    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" containerName="kube-rbac-proxy"
Sep 15 18:53:19.852560 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:53:19.851936    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406\": container with ID starting with 6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406 not found: ID does not exist" containerID="6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406"
Sep 15 18:53:19.852560 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851952    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406"} err="failed to get container status \"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406\": rpc error: code = NotFound desc = could not find container \"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406\": container with ID starting with 6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406 not found: ID does not exist"
Sep 15 18:53:19.852560 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.851964    2555 scope.go:117] "RemoveContainer" containerID="64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe"
Sep 15 18:53:19.852560 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.852214    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe"} err="failed to get container status \"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe\": rpc error: code = NotFound desc = could not find container \"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe\": container with ID starting with 64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe not found: ID does not exist"
Sep 15 18:53:19.852560 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.852235    2555 scope.go:117] "RemoveContainer" containerID="8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9"
Sep 15 18:53:19.852560 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.852518    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9"} err="failed to get container status \"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9\": rpc error: code = NotFound desc = could not find container \"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9\": container with ID starting with 8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9 not found: ID does not exist"
Sep 15 18:53:19.852560 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.852540    2555 scope.go:117] "RemoveContainer" containerID="801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb"
Sep 15 18:53:19.852907 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.852804    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb"} err="failed to get container status \"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb\": rpc error: code = NotFound desc = could not find container \"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb\": container with ID starting with 801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb not found: ID does not exist"
Sep 15 18:53:19.852907 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.852819    2555 scope.go:117] "RemoveContainer" containerID="aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6"
Sep 15 18:53:19.853037 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.853018    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6"} err="failed to get container status \"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6\": rpc error: code = NotFound desc = could not find container \"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6\": container with ID starting with aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6 not found: ID does not exist"
Sep 15 18:53:19.853077 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.853038    2555 scope.go:117] "RemoveContainer" containerID="35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda"
Sep 15 18:53:19.853340 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.853321    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda"} err="failed to get container status \"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda\": rpc error: code = NotFound desc = could not find container \"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda\": container with ID starting with 35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda not found: ID does not exist"
Sep 15 18:53:19.853383 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.853354    2555 scope.go:117] "RemoveContainer" containerID="f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2"
Sep 15 18:53:19.853761 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.853739    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2"} err="failed to get container status \"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2\": rpc error: code = NotFound desc = could not find container \"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2\": container with ID starting with f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2 not found: ID does not exist"
Sep 15 18:53:19.853761 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.853759    2555 scope.go:117] "RemoveContainer" containerID="6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406"
Sep 15 18:53:19.854062 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.854020    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406"} err="failed to get container status \"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406\": rpc error: code = NotFound desc = could not find container \"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406\": container with ID starting with 6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406 not found: ID does not exist"
Sep 15 18:53:19.854062 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.854046    2555 scope.go:117] "RemoveContainer" containerID="64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe"
Sep 15 18:53:19.854570 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.854343    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe"} err="failed to get container status \"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe\": rpc error: code = NotFound desc = could not find container \"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe\": container with ID starting with 64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe not found: ID does not exist"
Sep 15 18:53:19.854570 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.854381    2555 scope.go:117] "RemoveContainer" containerID="8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9"
Sep 15 18:53:19.855370 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.855109    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9"} err="failed to get container status \"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9\": rpc error: code = NotFound desc = could not find container \"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9\": container with ID starting with 8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9 not found: ID does not exist"
Sep 15 18:53:19.855370 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.855134    2555 scope.go:117] "RemoveContainer" containerID="801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb"
Sep 15 18:53:19.856186 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.856161    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb"} err="failed to get container status \"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb\": rpc error: code = NotFound desc = could not find container \"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb\": container with ID starting with 801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb not found: ID does not exist"
Sep 15 18:53:19.856285 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.856185    2555 scope.go:117] "RemoveContainer" containerID="aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6"
Sep 15 18:53:19.856481 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.856459    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6"} err="failed to get container status \"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6\": rpc error: code = NotFound desc = could not find container \"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6\": container with ID starting with aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6 not found: ID does not exist"
Sep 15 18:53:19.856481 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.856480    2555 scope.go:117] "RemoveContainer" containerID="35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda"
Sep 15 18:53:19.856869 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.856846    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda"} err="failed to get container status \"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda\": rpc error: code = NotFound desc = could not find container \"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda\": container with ID starting with 35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda not found: ID does not exist"
Sep 15 18:53:19.856928 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.856877    2555 scope.go:117] "RemoveContainer" containerID="f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2"
Sep 15 18:53:19.857068 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.857053    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.857124 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.857104    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2"} err="failed to get container status \"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2\": rpc error: code = NotFound desc = could not find container \"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2\": container with ID starting with f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2 not found: ID does not exist"
Sep 15 18:53:19.857160 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.857126    2555 scope.go:117] "RemoveContainer" containerID="6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406"
Sep 15 18:53:19.857353 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.857331    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406"} err="failed to get container status \"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406\": rpc error: code = NotFound desc = could not find container \"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406\": container with ID starting with 6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406 not found: ID does not exist"
Sep 15 18:53:19.857431 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.857355    2555 scope.go:117] "RemoveContainer" containerID="64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe"
Sep 15 18:53:19.857654 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.857632    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe"} err="failed to get container status \"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe\": rpc error: code = NotFound desc = could not find container \"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe\": container with ID starting with 64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe not found: ID does not exist"
Sep 15 18:53:19.857654 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.857652    2555 scope.go:117] "RemoveContainer" containerID="8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9"
Sep 15 18:53:19.857934 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.857912    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9"} err="failed to get container status \"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9\": rpc error: code = NotFound desc = could not find container \"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9\": container with ID starting with 8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9 not found: ID does not exist"
Sep 15 18:53:19.857989 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.857938    2555 scope.go:117] "RemoveContainer" containerID="801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb"
Sep 15 18:53:19.858181 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.858158    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb"} err="failed to get container status \"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb\": rpc error: code = NotFound desc = could not find container \"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb\": container with ID starting with 801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb not found: ID does not exist"
Sep 15 18:53:19.858235 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.858184    2555 scope.go:117] "RemoveContainer" containerID="aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6"
Sep 15 18:53:19.858465 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.858448    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6"} err="failed to get container status \"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6\": rpc error: code = NotFound desc = could not find container \"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6\": container with ID starting with aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6 not found: ID does not exist"
Sep 15 18:53:19.858518 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.858487    2555 scope.go:117] "RemoveContainer" containerID="35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda"
Sep 15 18:53:19.858722 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.858703    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda"} err="failed to get container status \"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda\": rpc error: code = NotFound desc = could not find container \"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda\": container with ID starting with 35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda not found: ID does not exist"
Sep 15 18:53:19.858795 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.858724    2555 scope.go:117] "RemoveContainer" containerID="f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2"
Sep 15 18:53:19.859012 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.858983    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2"} err="failed to get container status \"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2\": rpc error: code = NotFound desc = could not find container \"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2\": container with ID starting with f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2 not found: ID does not exist"
Sep 15 18:53:19.859012 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.859008    2555 scope.go:117] "RemoveContainer" containerID="6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406"
Sep 15 18:53:19.859392 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.859341    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406"} err="failed to get container status \"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406\": rpc error: code = NotFound desc = could not find container \"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406\": container with ID starting with 6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406 not found: ID does not exist"
Sep 15 18:53:19.859392 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.859391    2555 scope.go:117] "RemoveContainer" containerID="64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe"
Sep 15 18:53:19.859656 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.859638    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 18:53:19.859764 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.859654    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3eiolerg054sr\""
Sep 15 18:53:19.859764 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.859675    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe"} err="failed to get container status \"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe\": rpc error: code = NotFound desc = could not find container \"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe\": container with ID starting with 64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe not found: ID does not exist"
Sep 15 18:53:19.859764 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.859702    2555 scope.go:117] "RemoveContainer" containerID="8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9"
Sep 15 18:53:19.859764 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.859731    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 18:53:19.859967 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.859658    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 18:53:19.860066 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.860049    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 18:53:19.860131 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.860077    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 18:53:19.860131 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.860083    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 18:53:19.860131 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.860122    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 18:53:19.860241 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.860129    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9"} err="failed to get container status \"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9\": rpc error: code = NotFound desc = could not find container \"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9\": container with ID starting with 8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9 not found: ID does not exist"
Sep 15 18:53:19.860241 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.860150    2555 scope.go:117] "RemoveContainer" containerID="801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb"
Sep 15 18:53:19.860241 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.860208    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 18:53:19.860376 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.860354    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 18:53:19.860426 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.860385    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bcfdz\""
Sep 15 18:53:19.860426 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.860395    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb"} err="failed to get container status \"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb\": rpc error: code = NotFound desc = could not find container \"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb\": container with ID starting with 801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb not found: ID does not exist"
Sep 15 18:53:19.860426 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.860412    2555 scope.go:117] "RemoveContainer" containerID="aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6"
Sep 15 18:53:19.860700 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.860674    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 18:53:19.860770 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.860738    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6"} err="failed to get container status \"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6\": rpc error: code = NotFound desc = could not find container \"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6\": container with ID starting with aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6 not found: ID does not exist"
Sep 15 18:53:19.860770 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.860761    2555 scope.go:117] "RemoveContainer" containerID="35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda"
Sep 15 18:53:19.860770 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.860682    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 18:53:19.860919 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.860844    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 18:53:19.861036 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.861001    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda"} err="failed to get container status \"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda\": rpc error: code = NotFound desc = could not find container \"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda\": container with ID starting with 35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda not found: ID does not exist"
Sep 15 18:53:19.861036 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.861024    2555 scope.go:117] "RemoveContainer" containerID="f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2"
Sep 15 18:53:19.861306 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.861279    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2"} err="failed to get container status \"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2\": rpc error: code = NotFound desc = could not find container \"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2\": container with ID starting with f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2 not found: ID does not exist"
Sep 15 18:53:19.861392 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.861306    2555 scope.go:117] "RemoveContainer" containerID="6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406"
Sep 15 18:53:19.861665 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.861570    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406"} err="failed to get container status \"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406\": rpc error: code = NotFound desc = could not find container \"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406\": container with ID starting with 6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406 not found: ID does not exist"
Sep 15 18:53:19.861665 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.861636    2555 scope.go:117] "RemoveContainer" containerID="64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe"
Sep 15 18:53:19.861973 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.861950    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe"} err="failed to get container status \"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe\": rpc error: code = NotFound desc = could not find container \"64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe\": container with ID starting with 64ae19ca5dcadd0e4303b344806101ae736c158fa75c293d0caadf234133acfe not found: ID does not exist"
Sep 15 18:53:19.862041 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.861974    2555 scope.go:117] "RemoveContainer" containerID="8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9"
Sep 15 18:53:19.862288 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.862261    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9"} err="failed to get container status \"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9\": rpc error: code = NotFound desc = could not find container \"8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9\": container with ID starting with 8aa36d262930112fce532399e54bf07113db5275d65975ca88edfcb6ad3768e9 not found: ID does not exist"
Sep 15 18:53:19.862376 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.862288    2555 scope.go:117] "RemoveContainer" containerID="801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb"
Sep 15 18:53:19.862662 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.862642    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb"} err="failed to get container status \"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb\": rpc error: code = NotFound desc = could not find container \"801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb\": container with ID starting with 801693d78c35d0aa96a4b257cf1321e32b64a396db723fd6fe8c7a364a4a6bcb not found: ID does not exist"
Sep 15 18:53:19.862750 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.862663    2555 scope.go:117] "RemoveContainer" containerID="aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6"
Sep 15 18:53:19.862981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.862964    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 18:53:19.863039 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.862966    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6"} err="failed to get container status \"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6\": rpc error: code = NotFound desc = could not find container \"aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6\": container with ID starting with aacd55b3870d7c3ade6f26fe6935c9a590c32382d5de95b8bdcb13dbbbbe09d6 not found: ID does not exist"
Sep 15 18:53:19.863039 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.863001    2555 scope.go:117] "RemoveContainer" containerID="35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda"
Sep 15 18:53:19.863272 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.863250    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda"} err="failed to get container status \"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda\": rpc error: code = NotFound desc = could not find container \"35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda\": container with ID starting with 35cfff6c23968d49097a5e8f71e4cad316ffa6acba461ac0ffac3e04b35abeda not found: ID does not exist"
Sep 15 18:53:19.863272 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.863270    2555 scope.go:117] "RemoveContainer" containerID="f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2"
Sep 15 18:53:19.863553 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.863533    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2"} err="failed to get container status \"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2\": rpc error: code = NotFound desc = could not find container \"f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2\": container with ID starting with f847cbd4584801fab95c8e762ac6c078adc8973ae9781e38c4ee327e3fa2c8b2 not found: ID does not exist"
Sep 15 18:53:19.863553 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.863553    2555 scope.go:117] "RemoveContainer" containerID="6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406"
Sep 15 18:53:19.863834 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.863815    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406"} err="failed to get container status \"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406\": rpc error: code = NotFound desc = could not find container \"6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406\": container with ID starting with 6d9a042ece79568e5c95262ddcf4b01953b41ca9038e22f19b86d3b7ac097406 not found: ID does not exist"
Sep 15 18:53:19.865201 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.865184    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 18:53:19.870095 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.870073    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 18:53:19.914393 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914361    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.914393 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914394    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-web-config\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.914596 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914415    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.914596 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914493    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.914596 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914519    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.914596 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914544    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.914596 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914561    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/63ee6cce-db30-41f7-bcb4-fd7d8d145013-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.914783 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914624    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.914783 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914650    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.914783 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914669    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.914783 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914689    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74gfg\" (UniqueName: \"kubernetes.io/projected/63ee6cce-db30-41f7-bcb4-fd7d8d145013-kube-api-access-74gfg\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.914783 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914718    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63ee6cce-db30-41f7-bcb4-fd7d8d145013-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.914783 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914752    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.914783 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914771    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.915030 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914786    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.915030 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914821    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-config\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.915030 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914847    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.915030 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914886    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.915030 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914910    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63ee6cce-db30-41f7-bcb4-fd7d8d145013-config-out\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:19.915030 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:19.914932    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.015736 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.015693    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63ee6cce-db30-41f7-bcb4-fd7d8d145013-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.015981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.015746    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.015981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.015781    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.015981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.015798    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.015981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.015817    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-config\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.015981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.015843    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.015981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.015866    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.015981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.015898    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63ee6cce-db30-41f7-bcb4-fd7d8d145013-config-out\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.015981 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.015924    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.016643 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.016587    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.016743 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.016640    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.016743 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.016691    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.016743 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.016726    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-web-config\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.016905 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.016740    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.016905 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.016760    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.016905 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.016800    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.016905 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.016828    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.016905 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.016866    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.016905 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.016891    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/63ee6cce-db30-41f7-bcb4-fd7d8d145013-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.017194 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.016938    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.017194 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.016968    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.017194 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.016996    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.017194 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.017020    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74gfg\" (UniqueName: \"kubernetes.io/projected/63ee6cce-db30-41f7-bcb4-fd7d8d145013-kube-api-access-74gfg\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.017757 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.017735    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.019324 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.018824    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63ee6cce-db30-41f7-bcb4-fd7d8d145013-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.019324 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.018899    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-config\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.019324 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.019144    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.019324 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.019165    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.019587 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.019371    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63ee6cce-db30-41f7-bcb4-fd7d8d145013-config-out\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.019732 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.019705    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.021010 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.020324    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.021010 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.020469    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-web-config\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.021010 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.020968    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.021216 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.021148    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/63ee6cce-db30-41f7-bcb4-fd7d8d145013-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.021389 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.021370    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.021884 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.021857    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.022059 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.022043    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.022314 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.022295    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63ee6cce-db30-41f7-bcb4-fd7d8d145013-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.023514 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.023494    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/63ee6cce-db30-41f7-bcb4-fd7d8d145013-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.024504 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.024487    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74gfg\" (UniqueName: \"kubernetes.io/projected/63ee6cce-db30-41f7-bcb4-fd7d8d145013-kube-api-access-74gfg\") pod \"prometheus-k8s-0\" (UID: \"63ee6cce-db30-41f7-bcb4-fd7d8d145013\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.168084 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.167994    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:53:20.459479 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.459455    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 18:53:20.462268 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:53:20.462240    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ee6cce_db30_41f7_bcb4_fd7d8d145013.slice/crio-27959e13286e2352d8eaedaad5b204e40800783cf79a3d15452c5d7ab41134ec WatchSource:0}: Error finding container 27959e13286e2352d8eaedaad5b204e40800783cf79a3d15452c5d7ab41134ec: Status 404 returned error can't find the container with id 27959e13286e2352d8eaedaad5b204e40800783cf79a3d15452c5d7ab41134ec
Sep 15 18:53:20.806621 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.806563    2555 generic.go:356] "Generic (PLEG): container finished" podID="63ee6cce-db30-41f7-bcb4-fd7d8d145013" containerID="0f8074b46d1168202b664cca6e918df73c8a54e73393c58311b49e6dbec17234" exitCode=0
Sep 15 18:53:20.807153 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.806644    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63ee6cce-db30-41f7-bcb4-fd7d8d145013","Type":"ContainerDied","Data":"0f8074b46d1168202b664cca6e918df73c8a54e73393c58311b49e6dbec17234"}
Sep 15 18:53:20.807153 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:20.806711    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63ee6cce-db30-41f7-bcb4-fd7d8d145013","Type":"ContainerStarted","Data":"27959e13286e2352d8eaedaad5b204e40800783cf79a3d15452c5d7ab41134ec"}
Sep 15 18:53:21.040659 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:21.040622    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41" path="/var/lib/kubelet/pods/97df3fa8-4d31-4d31-aa7d-9b4e6a83cb41/volumes"
Sep 15 18:53:21.814015 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:21.813910    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52" event={"ID":"695b920f-2498-4cc6-8d0e-1095adfb7f2a","Type":"ContainerStarted","Data":"f2c96abaa52654d21667b2e3c461619b099c624231fbd303fcf585411fa09c1e"}
Sep 15 18:53:21.814015 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:21.813979    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52" event={"ID":"695b920f-2498-4cc6-8d0e-1095adfb7f2a","Type":"ContainerStarted","Data":"f20c8522f41444bb96cf637fed17f4bd837233703fe7fb046d30275bfc3fe500"}
Sep 15 18:53:21.814015 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:21.814009    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52" event={"ID":"695b920f-2498-4cc6-8d0e-1095adfb7f2a","Type":"ContainerStarted","Data":"21a037537df6267912531c8373de3076ea58af93ea6849315a49bd7265c8d52d"}
Sep 15 18:53:21.817444 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:21.817414    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63ee6cce-db30-41f7-bcb4-fd7d8d145013","Type":"ContainerStarted","Data":"b8cc07c622edc1585a0d6468c06d0d8172e66aec635edfdfc42f522c93129846"}
Sep 15 18:53:21.817592 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:21.817472    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63ee6cce-db30-41f7-bcb4-fd7d8d145013","Type":"ContainerStarted","Data":"1f7737f99ed89e3427be106cb0010c02228ae63c319cdd899372166a298cb4b6"}
Sep 15 18:53:21.817592 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:21.817507    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63ee6cce-db30-41f7-bcb4-fd7d8d145013","Type":"ContainerStarted","Data":"b3b1e0ff6f239ac7704c445956cd197fb66aa67155c63014e861f64bb2c7c948"}
Sep 15 18:53:21.817592 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:21.817532    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63ee6cce-db30-41f7-bcb4-fd7d8d145013","Type":"ContainerStarted","Data":"723991e643ca85aace64c499cacddbd936c61785f01201fa00b7a5f8873c7ae0"}
Sep 15 18:53:21.817592 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:21.817562    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63ee6cce-db30-41f7-bcb4-fd7d8d145013","Type":"ContainerStarted","Data":"e45965c94f431502c4a5df4aaf3c9eb40b461f0ecb3255c81afd37502bc37a13"}
Sep 15 18:53:21.817592 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:21.817592    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"63ee6cce-db30-41f7-bcb4-fd7d8d145013","Type":"ContainerStarted","Data":"2235cd15153e7ee41c449b71b78119fab7515d67249ee3f405f446da9295a07e"}
Sep 15 18:53:21.834716 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:21.834661    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-8545d8745b-fkp52" podStartSLOduration=1.695120156 podStartE2EDuration="3.834628088s" totalImagesPullingTime="2.139507932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:53:21.833199816 +0000 UTC m=+143.320528804" watchObservedRunningTime="2026-09-15 18:53:21.834628088 +0000 UTC m=+143.321957067"
Sep 15 18:53:21.859977 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:21.859926    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.836945098 podStartE2EDuration="2.859910719s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.965621ms" isStatefulPod=false podCreationTimestamp="2026-09-15 18:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:53:21.857791796 +0000 UTC m=+143.345120782" watchObservedRunningTime="2026-09-15 18:53:21.859910719 +0000 UTC m=+143.347239704"
Sep 15 18:53:25.168523 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:53:25.168482    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:54:20.168241 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:54:20.168184    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:54:20.183483 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:54:20.183457    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:54:21.063670 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:54:21.063643    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 18:55:58.945833 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:55:58.945802    2555 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 18:56:06.003418 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:06.003388    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kcrkx"]
Sep 15 18:56:06.006589 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:06.006574    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kcrkx"
Sep 15 18:56:06.008891 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:06.008867    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 15 18:56:06.008891 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:06.008882    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 15 18:56:06.009576 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:06.009553    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-s574v\""
Sep 15 18:56:06.015504 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:06.015467    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kcrkx"]
Sep 15 18:56:06.108123 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:06.108094    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00312c46-08f1-47ec-9545-e38f195e7d6a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kcrkx\" (UID: \"00312c46-08f1-47ec-9545-e38f195e7d6a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kcrkx"
Sep 15 18:56:06.108277 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:06.108159    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkk5j\" (UniqueName: \"kubernetes.io/projected/00312c46-08f1-47ec-9545-e38f195e7d6a-kube-api-access-vkk5j\") pod \"cert-manager-webhook-6486ffd997-kcrkx\" (UID: \"00312c46-08f1-47ec-9545-e38f195e7d6a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kcrkx"
Sep 15 18:56:06.209628 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:06.209557    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00312c46-08f1-47ec-9545-e38f195e7d6a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kcrkx\" (UID: \"00312c46-08f1-47ec-9545-e38f195e7d6a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kcrkx"
Sep 15 18:56:06.209818 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:06.209675    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkk5j\" (UniqueName: \"kubernetes.io/projected/00312c46-08f1-47ec-9545-e38f195e7d6a-kube-api-access-vkk5j\") pod \"cert-manager-webhook-6486ffd997-kcrkx\" (UID: \"00312c46-08f1-47ec-9545-e38f195e7d6a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kcrkx"
Sep 15 18:56:06.220132 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:06.220108    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00312c46-08f1-47ec-9545-e38f195e7d6a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kcrkx\" (UID: \"00312c46-08f1-47ec-9545-e38f195e7d6a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kcrkx"
Sep 15 18:56:06.220249 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:06.220187    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkk5j\" (UniqueName: \"kubernetes.io/projected/00312c46-08f1-47ec-9545-e38f195e7d6a-kube-api-access-vkk5j\") pod \"cert-manager-webhook-6486ffd997-kcrkx\" (UID: \"00312c46-08f1-47ec-9545-e38f195e7d6a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kcrkx"
Sep 15 18:56:06.329323 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:06.329258    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kcrkx"
Sep 15 18:56:06.604096 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:06.604068    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kcrkx"]
Sep 15 18:56:06.606909 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:56:06.606881    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00312c46_08f1_47ec_9545_e38f195e7d6a.slice/crio-8a8cee3b9ce72f0963eff1d70f1c63869d0cb60feb74a0d53006071ccd0cd10a WatchSource:0}: Error finding container 8a8cee3b9ce72f0963eff1d70f1c63869d0cb60feb74a0d53006071ccd0cd10a: Status 404 returned error can't find the container with id 8a8cee3b9ce72f0963eff1d70f1c63869d0cb60feb74a0d53006071ccd0cd10a
Sep 15 18:56:06.609166 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:06.609146    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 18:56:07.472077 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:07.472014    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kcrkx" event={"ID":"00312c46-08f1-47ec-9545-e38f195e7d6a","Type":"ContainerStarted","Data":"8a8cee3b9ce72f0963eff1d70f1c63869d0cb60feb74a0d53006071ccd0cd10a"}
Sep 15 18:56:10.488528 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:10.488494    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kcrkx" event={"ID":"00312c46-08f1-47ec-9545-e38f195e7d6a","Type":"ContainerStarted","Data":"1e4d9470c4bc7517780887b4d8a48d26180546c3263fff87a163867425523946"}
Sep 15 18:56:10.488938 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:10.488551    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kcrkx"
Sep 15 18:56:10.505026 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:10.504968    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-kcrkx" podStartSLOduration=2.344401182 podStartE2EDuration="5.5049496s" totalImagesPullingTime="3.160548418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:56:10.50387625 +0000 UTC m=+311.991205235" watchObservedRunningTime="2026-09-15 18:56:10.5049496 +0000 UTC m=+311.992278584"
Sep 15 18:56:16.494227 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:16.494193    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kcrkx"
Sep 15 18:56:24.723724 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:24.723690    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b5zgz"]
Sep 15 18:56:24.727008 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:24.726990    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b5zgz"
Sep 15 18:56:24.729262 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:24.729243    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:56:24.729922 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:24.729905    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 18:56:24.729995 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:24.729917    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-xjntx\""
Sep 15 18:56:24.734687 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:24.734666    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b5zgz"]
Sep 15 18:56:24.878879 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:24.878846    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkg48\" (UniqueName: \"kubernetes.io/projected/af171b3e-d379-4f04-9589-d43a1d33a9c9-kube-api-access-lkg48\") pod \"openshift-lws-operator-fd8ccff4c-b5zgz\" (UID: \"af171b3e-d379-4f04-9589-d43a1d33a9c9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b5zgz"
Sep 15 18:56:24.879047 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:24.878927    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af171b3e-d379-4f04-9589-d43a1d33a9c9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-b5zgz\" (UID: \"af171b3e-d379-4f04-9589-d43a1d33a9c9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b5zgz"
Sep 15 18:56:24.980377 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:24.980281    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkg48\" (UniqueName: \"kubernetes.io/projected/af171b3e-d379-4f04-9589-d43a1d33a9c9-kube-api-access-lkg48\") pod \"openshift-lws-operator-fd8ccff4c-b5zgz\" (UID: \"af171b3e-d379-4f04-9589-d43a1d33a9c9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b5zgz"
Sep 15 18:56:24.980527 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:24.980395    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af171b3e-d379-4f04-9589-d43a1d33a9c9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-b5zgz\" (UID: \"af171b3e-d379-4f04-9589-d43a1d33a9c9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b5zgz"
Sep 15 18:56:24.980740 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:24.980725    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af171b3e-d379-4f04-9589-d43a1d33a9c9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-b5zgz\" (UID: \"af171b3e-d379-4f04-9589-d43a1d33a9c9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b5zgz"
Sep 15 18:56:24.988749 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:24.988719    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkg48\" (UniqueName: \"kubernetes.io/projected/af171b3e-d379-4f04-9589-d43a1d33a9c9-kube-api-access-lkg48\") pod \"openshift-lws-operator-fd8ccff4c-b5zgz\" (UID: \"af171b3e-d379-4f04-9589-d43a1d33a9c9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b5zgz"
Sep 15 18:56:25.040378 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:25.040288    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b5zgz"
Sep 15 18:56:25.312186 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:25.312161    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b5zgz"]
Sep 15 18:56:25.314320 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:56:25.314288    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf171b3e_d379_4f04_9589_d43a1d33a9c9.slice/crio-cbe45b259a6c92206d2d3fb634a6e528b42ad5b6c225ee34cdbd0db420f9b8ea WatchSource:0}: Error finding container cbe45b259a6c92206d2d3fb634a6e528b42ad5b6c225ee34cdbd0db420f9b8ea: Status 404 returned error can't find the container with id cbe45b259a6c92206d2d3fb634a6e528b42ad5b6c225ee34cdbd0db420f9b8ea
Sep 15 18:56:25.548669 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:25.548633    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b5zgz" event={"ID":"af171b3e-d379-4f04-9589-d43a1d33a9c9","Type":"ContainerStarted","Data":"cbe45b259a6c92206d2d3fb634a6e528b42ad5b6c225ee34cdbd0db420f9b8ea"}
Sep 15 18:56:28.570119 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:28.570080    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b5zgz" event={"ID":"af171b3e-d379-4f04-9589-d43a1d33a9c9","Type":"ContainerStarted","Data":"beeb046b134315c2f0999e7991c643cd4eab7462179067294b3223ce667ec63d"}
Sep 15 18:56:28.585899 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:28.585848    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b5zgz" podStartSLOduration=2.062473186 podStartE2EDuration="4.585831328s" totalImagesPullingTime="2.523358142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:56:28.58392411 +0000 UTC m=+330.071253095" watchObservedRunningTime="2026-09-15 18:56:28.585831328 +0000 UTC m=+330.073160312"
Sep 15 18:56:44.765818 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.765787    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"]
Sep 15 18:56:44.770103 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.770087    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"
Sep 15 18:56:44.772874 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.772851    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 18:56:44.773033 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.773006    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h74qm\""
Sep 15 18:56:44.773226 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.773211    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 18:56:44.773287 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.773232    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 18:56:44.773287 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.773239    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 18:56:44.786085 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.786060    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"]
Sep 15 18:56:44.848542 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.848512    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfbll\" (UniqueName: \"kubernetes.io/projected/a5b931e3-1eb1-4272-b43c-29d12313a10a-kube-api-access-dfbll\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-crsfx\" (UID: \"a5b931e3-1eb1-4272-b43c-29d12313a10a\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"
Sep 15 18:56:44.848712 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.848562    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a5b931e3-1eb1-4272-b43c-29d12313a10a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-crsfx\" (UID: \"a5b931e3-1eb1-4272-b43c-29d12313a10a\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"
Sep 15 18:56:44.848712 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.848600    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a5b931e3-1eb1-4272-b43c-29d12313a10a-manifests\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-crsfx\" (UID: \"a5b931e3-1eb1-4272-b43c-29d12313a10a\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"
Sep 15 18:56:44.848712 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.848636    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5b931e3-1eb1-4272-b43c-29d12313a10a-tmp\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-crsfx\" (UID: \"a5b931e3-1eb1-4272-b43c-29d12313a10a\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"
Sep 15 18:56:44.848712 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.848653    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5b931e3-1eb1-4272-b43c-29d12313a10a-webhook-cert\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-crsfx\" (UID: \"a5b931e3-1eb1-4272-b43c-29d12313a10a\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"
Sep 15 18:56:44.949929 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.949898    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a5b931e3-1eb1-4272-b43c-29d12313a10a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-crsfx\" (UID: \"a5b931e3-1eb1-4272-b43c-29d12313a10a\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"
Sep 15 18:56:44.950104 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.949941    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a5b931e3-1eb1-4272-b43c-29d12313a10a-manifests\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-crsfx\" (UID: \"a5b931e3-1eb1-4272-b43c-29d12313a10a\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"
Sep 15 18:56:44.950104 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.949968    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5b931e3-1eb1-4272-b43c-29d12313a10a-tmp\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-crsfx\" (UID: \"a5b931e3-1eb1-4272-b43c-29d12313a10a\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"
Sep 15 18:56:44.950104 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.949988    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5b931e3-1eb1-4272-b43c-29d12313a10a-webhook-cert\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-crsfx\" (UID: \"a5b931e3-1eb1-4272-b43c-29d12313a10a\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"
Sep 15 18:56:44.950349 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.950165    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfbll\" (UniqueName: \"kubernetes.io/projected/a5b931e3-1eb1-4272-b43c-29d12313a10a-kube-api-access-dfbll\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-crsfx\" (UID: \"a5b931e3-1eb1-4272-b43c-29d12313a10a\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"
Sep 15 18:56:44.950349 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.950333    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a5b931e3-1eb1-4272-b43c-29d12313a10a-manifests\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-crsfx\" (UID: \"a5b931e3-1eb1-4272-b43c-29d12313a10a\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"
Sep 15 18:56:44.952500 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.952459    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5b931e3-1eb1-4272-b43c-29d12313a10a-tmp\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-crsfx\" (UID: \"a5b931e3-1eb1-4272-b43c-29d12313a10a\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"
Sep 15 18:56:44.952800 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.952782    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a5b931e3-1eb1-4272-b43c-29d12313a10a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-crsfx\" (UID: \"a5b931e3-1eb1-4272-b43c-29d12313a10a\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"
Sep 15 18:56:44.952965 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.952944    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5b931e3-1eb1-4272-b43c-29d12313a10a-webhook-cert\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-crsfx\" (UID: \"a5b931e3-1eb1-4272-b43c-29d12313a10a\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"
Sep 15 18:56:44.960422 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:44.960373    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfbll\" (UniqueName: \"kubernetes.io/projected/a5b931e3-1eb1-4272-b43c-29d12313a10a-kube-api-access-dfbll\") pod \"opendatahub-operator-controller-manager-7467f6b4cf-crsfx\" (UID: \"a5b931e3-1eb1-4272-b43c-29d12313a10a\") " pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"
Sep 15 18:56:45.080815 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:45.080731    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"
Sep 15 18:56:45.368386 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:45.368362    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"]
Sep 15 18:56:45.371065 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:56:45.371038    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b931e3_1eb1_4272_b43c_29d12313a10a.slice/crio-2ad52a0c4073771c641b9e2a950b83bca87f3214c214c77a85007c2e1da672b1 WatchSource:0}: Error finding container 2ad52a0c4073771c641b9e2a950b83bca87f3214c214c77a85007c2e1da672b1: Status 404 returned error can't find the container with id 2ad52a0c4073771c641b9e2a950b83bca87f3214c214c77a85007c2e1da672b1
Sep 15 18:56:45.646879 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:45.646789    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx" event={"ID":"a5b931e3-1eb1-4272-b43c-29d12313a10a","Type":"ContainerStarted","Data":"2ad52a0c4073771c641b9e2a950b83bca87f3214c214c77a85007c2e1da672b1"}
Sep 15 18:56:49.666886 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:49.666848    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx" event={"ID":"a5b931e3-1eb1-4272-b43c-29d12313a10a","Type":"ContainerStarted","Data":"0c382786f3206706a5e981f1d2baae7153e641b556f1f77854150f0791075411"}
Sep 15 18:56:50.673142 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:50.673112    2555 generic.go:356] "Generic (PLEG): container finished" podID="a5b931e3-1eb1-4272-b43c-29d12313a10a" containerID="0c382786f3206706a5e981f1d2baae7153e641b556f1f77854150f0791075411" exitCode=0
Sep 15 18:56:50.673566 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:50.673176    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx" event={"ID":"a5b931e3-1eb1-4272-b43c-29d12313a10a","Type":"ContainerDied","Data":"0c382786f3206706a5e981f1d2baae7153e641b556f1f77854150f0791075411"}
Sep 15 18:56:51.679837 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:51.679803    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx" event={"ID":"a5b931e3-1eb1-4272-b43c-29d12313a10a","Type":"ContainerStarted","Data":"f77d697e803309a955e3a2487f3e07fe6001c4120a0af0d20f0d385d0fb48e76"}
Sep 15 18:56:51.680261 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:51.679930    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"
Sep 15 18:56:51.701181 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:56:51.701137    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx" podStartSLOduration=2.144353971 podStartE2EDuration="7.701120174s" totalImagesPullingTime="4.52427829s" totalInitContainerRuntime="1.032487913s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:56:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:56:51.698885587 +0000 UTC m=+353.186214571" watchObservedRunningTime="2026-09-15 18:56:51.701120174 +0000 UTC m=+353.188449158"
Sep 15 18:57:02.685908 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:02.685876    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7467f6b4cf-crsfx"
Sep 15 18:57:04.047671 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.047639    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj"]
Sep 15 18:57:04.056449 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.056428    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj"
Sep 15 18:57:04.060413 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.060388    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-z9746\""
Sep 15 18:57:04.060548 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.060440    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 15 18:57:04.060548 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.060478    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 15 18:57:04.060548 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.060486    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 15 18:57:04.069431 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.069409    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj"]
Sep 15 18:57:04.118054 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.118026    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4fb0e9a9-273b-455e-9f54-227f9e19c380-manager-config\") pod \"lws-controller-manager-5cb45c66cf-w9dtj\" (UID: \"4fb0e9a9-273b-455e-9f54-227f9e19c380\") " pod="openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj"
Sep 15 18:57:04.118180 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.118057    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fb0e9a9-273b-455e-9f54-227f9e19c380-metrics-cert\") pod \"lws-controller-manager-5cb45c66cf-w9dtj\" (UID: \"4fb0e9a9-273b-455e-9f54-227f9e19c380\") " pod="openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj"
Sep 15 18:57:04.118180 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.118093    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fb0e9a9-273b-455e-9f54-227f9e19c380-cert\") pod \"lws-controller-manager-5cb45c66cf-w9dtj\" (UID: \"4fb0e9a9-273b-455e-9f54-227f9e19c380\") " pod="openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj"
Sep 15 18:57:04.118180 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.118159    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmj5p\" (UniqueName: \"kubernetes.io/projected/4fb0e9a9-273b-455e-9f54-227f9e19c380-kube-api-access-mmj5p\") pod \"lws-controller-manager-5cb45c66cf-w9dtj\" (UID: \"4fb0e9a9-273b-455e-9f54-227f9e19c380\") " pod="openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj"
Sep 15 18:57:04.219574 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.219544    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fb0e9a9-273b-455e-9f54-227f9e19c380-cert\") pod \"lws-controller-manager-5cb45c66cf-w9dtj\" (UID: \"4fb0e9a9-273b-455e-9f54-227f9e19c380\") " pod="openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj"
Sep 15 18:57:04.219757 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.219586    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmj5p\" (UniqueName: \"kubernetes.io/projected/4fb0e9a9-273b-455e-9f54-227f9e19c380-kube-api-access-mmj5p\") pod \"lws-controller-manager-5cb45c66cf-w9dtj\" (UID: \"4fb0e9a9-273b-455e-9f54-227f9e19c380\") " pod="openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj"
Sep 15 18:57:04.219757 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.219670    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4fb0e9a9-273b-455e-9f54-227f9e19c380-manager-config\") pod \"lws-controller-manager-5cb45c66cf-w9dtj\" (UID: \"4fb0e9a9-273b-455e-9f54-227f9e19c380\") " pod="openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj"
Sep 15 18:57:04.219757 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.219690    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fb0e9a9-273b-455e-9f54-227f9e19c380-metrics-cert\") pod \"lws-controller-manager-5cb45c66cf-w9dtj\" (UID: \"4fb0e9a9-273b-455e-9f54-227f9e19c380\") " pod="openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj"
Sep 15 18:57:04.220299 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.220275    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4fb0e9a9-273b-455e-9f54-227f9e19c380-manager-config\") pod \"lws-controller-manager-5cb45c66cf-w9dtj\" (UID: \"4fb0e9a9-273b-455e-9f54-227f9e19c380\") " pod="openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj"
Sep 15 18:57:04.222266 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.222236    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fb0e9a9-273b-455e-9f54-227f9e19c380-metrics-cert\") pod \"lws-controller-manager-5cb45c66cf-w9dtj\" (UID: \"4fb0e9a9-273b-455e-9f54-227f9e19c380\") " pod="openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj"
Sep 15 18:57:04.222356 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.222288    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fb0e9a9-273b-455e-9f54-227f9e19c380-cert\") pod \"lws-controller-manager-5cb45c66cf-w9dtj\" (UID: \"4fb0e9a9-273b-455e-9f54-227f9e19c380\") " pod="openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj"
Sep 15 18:57:04.240993 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.240971    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmj5p\" (UniqueName: \"kubernetes.io/projected/4fb0e9a9-273b-455e-9f54-227f9e19c380-kube-api-access-mmj5p\") pod \"lws-controller-manager-5cb45c66cf-w9dtj\" (UID: \"4fb0e9a9-273b-455e-9f54-227f9e19c380\") " pod="openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj"
Sep 15 18:57:04.365648 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.365535    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj"
Sep 15 18:57:04.644347 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.644326    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj"]
Sep 15 18:57:04.646654 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:57:04.646625    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb0e9a9_273b_455e_9f54_227f9e19c380.slice/crio-1546e9adaa83c9cc1b7cd303a9f6daedd2ab69ef1cf7c4c3b8e69e73e1054326 WatchSource:0}: Error finding container 1546e9adaa83c9cc1b7cd303a9f6daedd2ab69ef1cf7c4c3b8e69e73e1054326: Status 404 returned error can't find the container with id 1546e9adaa83c9cc1b7cd303a9f6daedd2ab69ef1cf7c4c3b8e69e73e1054326
Sep 15 18:57:04.703929 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.703900    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5b76b67bf4-xtqs9"]
Sep 15 18:57:04.708430 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.708412    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-xtqs9"
Sep 15 18:57:04.710764 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.710742    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-w2hsw\""
Sep 15 18:57:04.710877 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.710767    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 18:57:04.711529 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.711514    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 18:57:04.712492 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.712477    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 18:57:04.712813 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.712797    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 18:57:04.723818 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.723798    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b76b67bf4-xtqs9"]
Sep 15 18:57:04.741776 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.741744    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj" event={"ID":"4fb0e9a9-273b-455e-9f54-227f9e19c380","Type":"ContainerStarted","Data":"1546e9adaa83c9cc1b7cd303a9f6daedd2ab69ef1cf7c4c3b8e69e73e1054326"}
Sep 15 18:57:04.825259 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.825226    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/240e4169-e142-49f2-9f8d-110075a03bc6-tls-certs\") pod \"kube-auth-proxy-5b76b67bf4-xtqs9\" (UID: \"240e4169-e142-49f2-9f8d-110075a03bc6\") " pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-xtqs9"
Sep 15 18:57:04.825410 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.825294    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xhz4\" (UniqueName: \"kubernetes.io/projected/240e4169-e142-49f2-9f8d-110075a03bc6-kube-api-access-9xhz4\") pod \"kube-auth-proxy-5b76b67bf4-xtqs9\" (UID: \"240e4169-e142-49f2-9f8d-110075a03bc6\") " pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-xtqs9"
Sep 15 18:57:04.825410 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.825331    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/240e4169-e142-49f2-9f8d-110075a03bc6-tmp\") pod \"kube-auth-proxy-5b76b67bf4-xtqs9\" (UID: \"240e4169-e142-49f2-9f8d-110075a03bc6\") " pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-xtqs9"
Sep 15 18:57:04.926071 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.925985    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/240e4169-e142-49f2-9f8d-110075a03bc6-tmp\") pod \"kube-auth-proxy-5b76b67bf4-xtqs9\" (UID: \"240e4169-e142-49f2-9f8d-110075a03bc6\") " pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-xtqs9"
Sep 15 18:57:04.926215 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.926081    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/240e4169-e142-49f2-9f8d-110075a03bc6-tls-certs\") pod \"kube-auth-proxy-5b76b67bf4-xtqs9\" (UID: \"240e4169-e142-49f2-9f8d-110075a03bc6\") " pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-xtqs9"
Sep 15 18:57:04.926215 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.926143    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xhz4\" (UniqueName: \"kubernetes.io/projected/240e4169-e142-49f2-9f8d-110075a03bc6-kube-api-access-9xhz4\") pod \"kube-auth-proxy-5b76b67bf4-xtqs9\" (UID: \"240e4169-e142-49f2-9f8d-110075a03bc6\") " pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-xtqs9"
Sep 15 18:57:04.928240 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.928216    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/240e4169-e142-49f2-9f8d-110075a03bc6-tmp\") pod \"kube-auth-proxy-5b76b67bf4-xtqs9\" (UID: \"240e4169-e142-49f2-9f8d-110075a03bc6\") " pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-xtqs9"
Sep 15 18:57:04.928434 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.928416    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/240e4169-e142-49f2-9f8d-110075a03bc6-tls-certs\") pod \"kube-auth-proxy-5b76b67bf4-xtqs9\" (UID: \"240e4169-e142-49f2-9f8d-110075a03bc6\") " pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-xtqs9"
Sep 15 18:57:04.933915 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:04.933898    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xhz4\" (UniqueName: \"kubernetes.io/projected/240e4169-e142-49f2-9f8d-110075a03bc6-kube-api-access-9xhz4\") pod \"kube-auth-proxy-5b76b67bf4-xtqs9\" (UID: \"240e4169-e142-49f2-9f8d-110075a03bc6\") " pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-xtqs9"
Sep 15 18:57:05.018027 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:05.017997    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-xtqs9"
Sep 15 18:57:05.293645 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:05.289244    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b76b67bf4-xtqs9"]
Sep 15 18:57:05.752600 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:05.752547    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-xtqs9" event={"ID":"240e4169-e142-49f2-9f8d-110075a03bc6","Type":"ContainerStarted","Data":"a72f5e45006ea0689f21926f42e1a21c9c5feae9f700aafc139e49af3f42d3fd"}
Sep 15 18:57:06.761596 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:06.761552    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj" event={"ID":"4fb0e9a9-273b-455e-9f54-227f9e19c380","Type":"ContainerStarted","Data":"5ceee333756dbb3d11c9152550abcb65e95dbda541e4baa8c04440980723fda4"}
Sep 15 18:57:06.762060 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:06.761781    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj"
Sep 15 18:57:06.781504 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:06.781449    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj" podStartSLOduration=0.828766774 podStartE2EDuration="2.781428581s" totalImagesPullingTime="1.952661807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:57:06.778282273 +0000 UTC m=+368.265611265" watchObservedRunningTime="2026-09-15 18:57:06.781428581 +0000 UTC m=+368.268757569"
Sep 15 18:57:08.772358 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:08.772321    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-xtqs9" event={"ID":"240e4169-e142-49f2-9f8d-110075a03bc6","Type":"ContainerStarted","Data":"b382d7e5cdcfb795a849e2757cc31743e7650253067fc3b8dc812c93a7a05017"}
Sep 15 18:57:08.787903 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:08.787855    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5b76b67bf4-xtqs9" podStartSLOduration=1.817687135 podStartE2EDuration="4.787837895s" totalImagesPullingTime="2.97015076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:57:08.787010525 +0000 UTC m=+370.274339511" watchObservedRunningTime="2026-09-15 18:57:08.787837895 +0000 UTC m=+370.275166889"
Sep 15 18:57:17.769463 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:57:17.769430    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5cb45c66cf-w9dtj"
Sep 15 18:58:41.048137 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:41.048106    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-x9lmq"]
Sep 15 18:58:41.051451 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:41.051422    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-x9lmq"
Sep 15 18:58:41.053811 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:41.053788    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 18:58:41.053811 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:41.053807    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 15 18:58:41.053986 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:41.053828    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 18:58:41.053986 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:41.053967    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vmhxn\""
Sep 15 18:58:41.065154 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:41.065130    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-x9lmq"]
Sep 15 18:58:41.171510 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:41.171475    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2zsv\" (UniqueName: \"kubernetes.io/projected/2779bd3b-4cdb-476b-9d11-af449b40d819-kube-api-access-h2zsv\") pod \"dns-operator-controller-manager-7b84cd7c9c-x9lmq\" (UID: \"2779bd3b-4cdb-476b-9d11-af449b40d819\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-x9lmq"
Sep 15 18:58:41.272083 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:41.272047    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2zsv\" (UniqueName: \"kubernetes.io/projected/2779bd3b-4cdb-476b-9d11-af449b40d819-kube-api-access-h2zsv\") pod \"dns-operator-controller-manager-7b84cd7c9c-x9lmq\" (UID: \"2779bd3b-4cdb-476b-9d11-af449b40d819\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-x9lmq"
Sep 15 18:58:41.280430 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:41.280399    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2zsv\" (UniqueName: \"kubernetes.io/projected/2779bd3b-4cdb-476b-9d11-af449b40d819-kube-api-access-h2zsv\") pod \"dns-operator-controller-manager-7b84cd7c9c-x9lmq\" (UID: \"2779bd3b-4cdb-476b-9d11-af449b40d819\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-x9lmq"
Sep 15 18:58:41.362583 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:41.362500    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-x9lmq"
Sep 15 18:58:41.637126 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:41.637046    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-x9lmq"]
Sep 15 18:58:41.640568 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:58:41.640545    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2779bd3b_4cdb_476b_9d11_af449b40d819.slice/crio-7ec3a41b03ff10315e8329b1bf157744b117dc389dcfb70764b9191062cf6a26 WatchSource:0}: Error finding container 7ec3a41b03ff10315e8329b1bf157744b117dc389dcfb70764b9191062cf6a26: Status 404 returned error can't find the container with id 7ec3a41b03ff10315e8329b1bf157744b117dc389dcfb70764b9191062cf6a26
Sep 15 18:58:42.204602 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:42.204565    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-x9lmq" event={"ID":"2779bd3b-4cdb-476b-9d11-af449b40d819","Type":"ContainerStarted","Data":"7ec3a41b03ff10315e8329b1bf157744b117dc389dcfb70764b9191062cf6a26"}
Sep 15 18:58:43.712471 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:43.712433    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-ldbhc"]
Sep 15 18:58:43.715695 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:43.715674    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ldbhc"
Sep 15 18:58:43.717899 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:43.717858    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ncz6z\""
Sep 15 18:58:43.718022 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:43.717895    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 15 18:58:43.718022 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:43.717896    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 15 18:58:43.723730 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:43.723704    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-ldbhc"]
Sep 15 18:58:43.795976 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:43.795943    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-ldbhc\" (UID: \"f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ldbhc"
Sep 15 18:58:43.796142 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:43.796017    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlkk5\" (UniqueName: \"kubernetes.io/projected/f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97-kube-api-access-jlkk5\") pod \"kuadrant-console-plugin-fbdd4f445-ldbhc\" (UID: \"f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ldbhc"
Sep 15 18:58:43.796142 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:43.796063    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-ldbhc\" (UID: \"f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ldbhc"
Sep 15 18:58:43.897307 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:43.897275    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-ldbhc\" (UID: \"f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ldbhc"
Sep 15 18:58:43.897486 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:43.897347    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-ldbhc\" (UID: \"f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ldbhc"
Sep 15 18:58:43.897486 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:43.897421    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlkk5\" (UniqueName: \"kubernetes.io/projected/f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97-kube-api-access-jlkk5\") pod \"kuadrant-console-plugin-fbdd4f445-ldbhc\" (UID: \"f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ldbhc"
Sep 15 18:58:43.898085 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:43.898061    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-ldbhc\" (UID: \"f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ldbhc"
Sep 15 18:58:43.899901 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:43.899878    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-ldbhc\" (UID: \"f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ldbhc"
Sep 15 18:58:43.905304 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:43.905278    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlkk5\" (UniqueName: \"kubernetes.io/projected/f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97-kube-api-access-jlkk5\") pod \"kuadrant-console-plugin-fbdd4f445-ldbhc\" (UID: \"f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ldbhc"
Sep 15 18:58:44.028099 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:44.028056    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ldbhc"
Sep 15 18:58:45.073046 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:45.072951    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-ldbhc"]
Sep 15 18:58:45.075864 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:58:45.075840    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4b5f5d7_9a02_4864_96f7_3cb99ecc7c97.slice/crio-3c091f2f070c4c209153c3d89294c6def84743329d30832fb60c2a15cdd455a2 WatchSource:0}: Error finding container 3c091f2f070c4c209153c3d89294c6def84743329d30832fb60c2a15cdd455a2: Status 404 returned error can't find the container with id 3c091f2f070c4c209153c3d89294c6def84743329d30832fb60c2a15cdd455a2
Sep 15 18:58:45.225879 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:45.225842    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-x9lmq" event={"ID":"2779bd3b-4cdb-476b-9d11-af449b40d819","Type":"ContainerStarted","Data":"e9adbbc50cdb94ee3e70f19a049440f15a81bc731009880727dc22d919b7243e"}
Sep 15 18:58:45.226067 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:45.225993    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-x9lmq"
Sep 15 18:58:45.227118 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:45.227092    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ldbhc" event={"ID":"f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97","Type":"ContainerStarted","Data":"3c091f2f070c4c209153c3d89294c6def84743329d30832fb60c2a15cdd455a2"}
Sep 15 18:58:45.242883 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:45.242829    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-x9lmq" podStartSLOduration=1.007312863 podStartE2EDuration="4.242805263s" totalImagesPullingTime="3.2354924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:58:45.242245576 +0000 UTC m=+466.729574560" watchObservedRunningTime="2026-09-15 18:58:45.242805263 +0000 UTC m=+466.730134253"
Sep 15 18:58:52.263556 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:52.263522    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ldbhc" event={"ID":"f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97","Type":"ContainerStarted","Data":"912e6873bf345417de8ac7c0424f8766abff991056dbe4de8f49c0c249bf918c"}
Sep 15 18:58:52.280740 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:52.280692    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ldbhc" podStartSLOduration=3.085151446 podStartE2EDuration="9.280677197s" totalImagesPullingTime="6.195525751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:58:52.278802438 +0000 UTC m=+473.766131424" watchObservedRunningTime="2026-09-15 18:58:52.280677197 +0000 UTC m=+473.768006182"
Sep 15 18:58:55.469019 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:55.468986    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn"]
Sep 15 18:58:55.472287 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:55.472269    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn"
Sep 15 18:58:55.475345 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:55.475323    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lhsr2\""
Sep 15 18:58:55.487643 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:55.487597    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn"]
Sep 15 18:58:55.613751 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:55.613720    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3038ea59-f062-4a17-b5a1-3655609b4ae3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j4chn\" (UID: \"3038ea59-f062-4a17-b5a1-3655609b4ae3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn"
Sep 15 18:58:55.613913 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:55.613758    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dsbb\" (UniqueName: \"kubernetes.io/projected/3038ea59-f062-4a17-b5a1-3655609b4ae3-kube-api-access-6dsbb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j4chn\" (UID: \"3038ea59-f062-4a17-b5a1-3655609b4ae3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn"
Sep 15 18:58:55.714280 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:55.714241    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3038ea59-f062-4a17-b5a1-3655609b4ae3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j4chn\" (UID: \"3038ea59-f062-4a17-b5a1-3655609b4ae3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn"
Sep 15 18:58:55.714280 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:55.714283    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dsbb\" (UniqueName: \"kubernetes.io/projected/3038ea59-f062-4a17-b5a1-3655609b4ae3-kube-api-access-6dsbb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j4chn\" (UID: \"3038ea59-f062-4a17-b5a1-3655609b4ae3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn"
Sep 15 18:58:55.714676 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:55.714655    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3038ea59-f062-4a17-b5a1-3655609b4ae3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j4chn\" (UID: \"3038ea59-f062-4a17-b5a1-3655609b4ae3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn"
Sep 15 18:58:55.727633 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:55.727562    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dsbb\" (UniqueName: \"kubernetes.io/projected/3038ea59-f062-4a17-b5a1-3655609b4ae3-kube-api-access-6dsbb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j4chn\" (UID: \"3038ea59-f062-4a17-b5a1-3655609b4ae3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn"
Sep 15 18:58:55.782148 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:55.782119    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn"
Sep 15 18:58:56.059575 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:56.059537    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn"]
Sep 15 18:58:56.061880 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:58:56.061859    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3038ea59_f062_4a17_b5a1_3655609b4ae3.slice/crio-a3b6741939e685401a904177aed61ffe0d7367cd65d4a6c60dda9e7aaddf22fb WatchSource:0}: Error finding container a3b6741939e685401a904177aed61ffe0d7367cd65d4a6c60dda9e7aaddf22fb: Status 404 returned error can't find the container with id a3b6741939e685401a904177aed61ffe0d7367cd65d4a6c60dda9e7aaddf22fb
Sep 15 18:58:56.234587 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:56.234557    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-x9lmq"
Sep 15 18:58:56.284008 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:58:56.283974    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn" event={"ID":"3038ea59-f062-4a17-b5a1-3655609b4ae3","Type":"ContainerStarted","Data":"a3b6741939e685401a904177aed61ffe0d7367cd65d4a6c60dda9e7aaddf22fb"}
Sep 15 18:59:00.309960 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:00.309915    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn" event={"ID":"3038ea59-f062-4a17-b5a1-3655609b4ae3","Type":"ContainerStarted","Data":"805fe67b7ab24a5a82d54f4027f2adb3c3e5727b30c90932998973bb3d8ec0b4"}
Sep 15 18:59:00.310376 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:00.310059    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn"
Sep 15 18:59:00.333092 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:00.333036    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn" podStartSLOduration=1.6271056270000002 podStartE2EDuration="5.333018052s" totalImagesPullingTime="3.705912425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:59:00.330768683 +0000 UTC m=+481.818097678" watchObservedRunningTime="2026-09-15 18:59:00.333018052 +0000 UTC m=+481.820347037"
Sep 15 18:59:11.317065 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:11.317035    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn"
Sep 15 18:59:33.233191 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:33.233160    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-bjtkn"]
Sep 15 18:59:33.235672 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:33.235650    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-bjtkn"
Sep 15 18:59:33.237829 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:33.237806    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8r4b2\""
Sep 15 18:59:33.246131 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:33.246104    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-bjtkn"]
Sep 15 18:59:33.350278 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:33.350242    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nkdl\" (UniqueName: \"kubernetes.io/projected/0a9c7f46-9f20-400a-bf54-dfd277e0dc5a-kube-api-access-6nkdl\") pod \"authorino-7bcfd8f9cd-bjtkn\" (UID: \"0a9c7f46-9f20-400a-bf54-dfd277e0dc5a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-bjtkn"
Sep 15 18:59:33.451724 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:33.451687    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nkdl\" (UniqueName: \"kubernetes.io/projected/0a9c7f46-9f20-400a-bf54-dfd277e0dc5a-kube-api-access-6nkdl\") pod \"authorino-7bcfd8f9cd-bjtkn\" (UID: \"0a9c7f46-9f20-400a-bf54-dfd277e0dc5a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-bjtkn"
Sep 15 18:59:33.459795 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:33.459765    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nkdl\" (UniqueName: \"kubernetes.io/projected/0a9c7f46-9f20-400a-bf54-dfd277e0dc5a-kube-api-access-6nkdl\") pod \"authorino-7bcfd8f9cd-bjtkn\" (UID: \"0a9c7f46-9f20-400a-bf54-dfd277e0dc5a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-bjtkn"
Sep 15 18:59:33.545301 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:33.545264    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-bjtkn"
Sep 15 18:59:33.811604 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:33.811522    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-bjtkn"]
Sep 15 18:59:33.815016 ip-10-0-132-28 kubenswrapper[2555]: W0915 18:59:33.814987    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a9c7f46_9f20_400a_bf54_dfd277e0dc5a.slice/crio-f6eef147dfef86ce90e3ebe6db98a529e79d59dcb756e2161df98f6c2d745e6f WatchSource:0}: Error finding container f6eef147dfef86ce90e3ebe6db98a529e79d59dcb756e2161df98f6c2d745e6f: Status 404 returned error can't find the container with id f6eef147dfef86ce90e3ebe6db98a529e79d59dcb756e2161df98f6c2d745e6f
Sep 15 18:59:34.474051 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:34.474015    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-bjtkn" event={"ID":"0a9c7f46-9f20-400a-bf54-dfd277e0dc5a","Type":"ContainerStarted","Data":"f6eef147dfef86ce90e3ebe6db98a529e79d59dcb756e2161df98f6c2d745e6f"}
Sep 15 18:59:36.441588 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:36.441534    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-bjtkn"]
Sep 15 18:59:36.486018 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:36.485983    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-bjtkn" event={"ID":"0a9c7f46-9f20-400a-bf54-dfd277e0dc5a","Type":"ContainerStarted","Data":"ad2bbfd952b3239179af1e0c83fd749c091227b908700fb5863a2c04839cd077"}
Sep 15 18:59:36.499111 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:36.499053    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-bjtkn" podStartSLOduration=1.778788075 podStartE2EDuration="3.499034434s" totalImagesPullingTime="1.720246359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:59:36.498984741 +0000 UTC m=+517.986313726" watchObservedRunningTime="2026-09-15 18:59:36.499034434 +0000 UTC m=+517.986363419"
Sep 15 18:59:37.491089 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:37.491050    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-bjtkn" podUID="0a9c7f46-9f20-400a-bf54-dfd277e0dc5a" containerName="authorino" containerID="cri-o://ad2bbfd952b3239179af1e0c83fd749c091227b908700fb5863a2c04839cd077" gracePeriod=30
Sep 15 18:59:37.737220 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:37.737193    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-bjtkn"
Sep 15 18:59:37.890691 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:37.890655    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nkdl\" (UniqueName: \"kubernetes.io/projected/0a9c7f46-9f20-400a-bf54-dfd277e0dc5a-kube-api-access-6nkdl\") pod \"0a9c7f46-9f20-400a-bf54-dfd277e0dc5a\" (UID: \"0a9c7f46-9f20-400a-bf54-dfd277e0dc5a\") "
Sep 15 18:59:37.892848 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:37.892819    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9c7f46-9f20-400a-bf54-dfd277e0dc5a-kube-api-access-6nkdl" (OuterVolumeSpecName: "kube-api-access-6nkdl") pod "0a9c7f46-9f20-400a-bf54-dfd277e0dc5a" (UID: "0a9c7f46-9f20-400a-bf54-dfd277e0dc5a"). InnerVolumeSpecName "kube-api-access-6nkdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:59:37.991548 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:37.991507    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nkdl\" (UniqueName: \"kubernetes.io/projected/0a9c7f46-9f20-400a-bf54-dfd277e0dc5a-kube-api-access-6nkdl\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 18:59:38.497054 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:38.497023    2555 generic.go:356] "Generic (PLEG): container finished" podID="0a9c7f46-9f20-400a-bf54-dfd277e0dc5a" containerID="ad2bbfd952b3239179af1e0c83fd749c091227b908700fb5863a2c04839cd077" exitCode=0
Sep 15 18:59:38.497454 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:38.497064    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-bjtkn"
Sep 15 18:59:38.497454 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:38.497114    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-bjtkn" event={"ID":"0a9c7f46-9f20-400a-bf54-dfd277e0dc5a","Type":"ContainerDied","Data":"ad2bbfd952b3239179af1e0c83fd749c091227b908700fb5863a2c04839cd077"}
Sep 15 18:59:38.497454 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:38.497154    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-bjtkn" event={"ID":"0a9c7f46-9f20-400a-bf54-dfd277e0dc5a","Type":"ContainerDied","Data":"f6eef147dfef86ce90e3ebe6db98a529e79d59dcb756e2161df98f6c2d745e6f"}
Sep 15 18:59:38.497454 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:38.497172    2555 scope.go:117] "RemoveContainer" containerID="ad2bbfd952b3239179af1e0c83fd749c091227b908700fb5863a2c04839cd077"
Sep 15 18:59:38.508165 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:38.508146    2555 scope.go:117] "RemoveContainer" containerID="ad2bbfd952b3239179af1e0c83fd749c091227b908700fb5863a2c04839cd077"
Sep 15 18:59:38.508536 ip-10-0-132-28 kubenswrapper[2555]: E0915 18:59:38.508518    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad2bbfd952b3239179af1e0c83fd749c091227b908700fb5863a2c04839cd077\": container with ID starting with ad2bbfd952b3239179af1e0c83fd749c091227b908700fb5863a2c04839cd077 not found: ID does not exist" containerID="ad2bbfd952b3239179af1e0c83fd749c091227b908700fb5863a2c04839cd077"
Sep 15 18:59:38.508597 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:38.508543    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad2bbfd952b3239179af1e0c83fd749c091227b908700fb5863a2c04839cd077"} err="failed to get container status \"ad2bbfd952b3239179af1e0c83fd749c091227b908700fb5863a2c04839cd077\": rpc error: code = NotFound desc = could not find container \"ad2bbfd952b3239179af1e0c83fd749c091227b908700fb5863a2c04839cd077\": container with ID starting with ad2bbfd952b3239179af1e0c83fd749c091227b908700fb5863a2c04839cd077 not found: ID does not exist"
Sep 15 18:59:38.519908 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:38.519866    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-bjtkn"]
Sep 15 18:59:38.522226 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:38.522204    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-bjtkn"]
Sep 15 18:59:39.040737 ip-10-0-132-28 kubenswrapper[2555]: I0915 18:59:39.040710    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a9c7f46-9f20-400a-bf54-dfd277e0dc5a" path="/var/lib/kubelet/pods/0a9c7f46-9f20-400a-bf54-dfd277e0dc5a/volumes"
Sep 15 19:00:02.124473 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:02.124423    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66b5f6c4f-ldbrg"]
Sep 15 19:00:02.124869 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:02.124808    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9c7f46-9f20-400a-bf54-dfd277e0dc5a" containerName="authorino"
Sep 15 19:00:02.124869 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:02.124820    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9c7f46-9f20-400a-bf54-dfd277e0dc5a" containerName="authorino"
Sep 15 19:00:02.124939 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:02.124881    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a9c7f46-9f20-400a-bf54-dfd277e0dc5a" containerName="authorino"
Sep 15 19:00:02.129050 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:02.129031    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66b5f6c4f-ldbrg"
Sep 15 19:00:02.132267 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:02.132242    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8r4b2\""
Sep 15 19:00:02.132383 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:02.132293    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 19:00:02.137760 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:02.137733    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66b5f6c4f-ldbrg"]
Sep 15 19:00:02.220799 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:02.220768    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25430b78-a108-4ac2-8942-9ac8434b2370-tls-cert\") pod \"authorino-66b5f6c4f-ldbrg\" (UID: \"25430b78-a108-4ac2-8942-9ac8434b2370\") " pod="kuadrant-system/authorino-66b5f6c4f-ldbrg"
Sep 15 19:00:02.220799 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:02.220801    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk66x\" (UniqueName: \"kubernetes.io/projected/25430b78-a108-4ac2-8942-9ac8434b2370-kube-api-access-bk66x\") pod \"authorino-66b5f6c4f-ldbrg\" (UID: \"25430b78-a108-4ac2-8942-9ac8434b2370\") " pod="kuadrant-system/authorino-66b5f6c4f-ldbrg"
Sep 15 19:00:02.321948 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:02.321910    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25430b78-a108-4ac2-8942-9ac8434b2370-tls-cert\") pod \"authorino-66b5f6c4f-ldbrg\" (UID: \"25430b78-a108-4ac2-8942-9ac8434b2370\") " pod="kuadrant-system/authorino-66b5f6c4f-ldbrg"
Sep 15 19:00:02.321948 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:02.321949    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk66x\" (UniqueName: \"kubernetes.io/projected/25430b78-a108-4ac2-8942-9ac8434b2370-kube-api-access-bk66x\") pod \"authorino-66b5f6c4f-ldbrg\" (UID: \"25430b78-a108-4ac2-8942-9ac8434b2370\") " pod="kuadrant-system/authorino-66b5f6c4f-ldbrg"
Sep 15 19:00:02.324343 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:02.324326    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25430b78-a108-4ac2-8942-9ac8434b2370-tls-cert\") pod \"authorino-66b5f6c4f-ldbrg\" (UID: \"25430b78-a108-4ac2-8942-9ac8434b2370\") " pod="kuadrant-system/authorino-66b5f6c4f-ldbrg"
Sep 15 19:00:02.329311 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:02.329288    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk66x\" (UniqueName: \"kubernetes.io/projected/25430b78-a108-4ac2-8942-9ac8434b2370-kube-api-access-bk66x\") pod \"authorino-66b5f6c4f-ldbrg\" (UID: \"25430b78-a108-4ac2-8942-9ac8434b2370\") " pod="kuadrant-system/authorino-66b5f6c4f-ldbrg"
Sep 15 19:00:02.438321 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:02.438230    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66b5f6c4f-ldbrg"
Sep 15 19:00:02.707483 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:02.707458    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66b5f6c4f-ldbrg"]
Sep 15 19:00:02.709652 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:00:02.709625    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25430b78_a108_4ac2_8942_9ac8434b2370.slice/crio-d9d78ba6e878452d193b1fb8d94284983c445e91bbee6ec9edf82a0b5ae20ce7 WatchSource:0}: Error finding container d9d78ba6e878452d193b1fb8d94284983c445e91bbee6ec9edf82a0b5ae20ce7: Status 404 returned error can't find the container with id d9d78ba6e878452d193b1fb8d94284983c445e91bbee6ec9edf82a0b5ae20ce7
Sep 15 19:00:03.616415 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:03.616380    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66b5f6c4f-ldbrg" event={"ID":"25430b78-a108-4ac2-8942-9ac8434b2370","Type":"ContainerStarted","Data":"fb602fb71caa327701ab9c857c93423effc88aec9c669c489c9aa0a0462e124f"}
Sep 15 19:00:03.616809 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:03.616423    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66b5f6c4f-ldbrg" event={"ID":"25430b78-a108-4ac2-8942-9ac8434b2370","Type":"ContainerStarted","Data":"d9d78ba6e878452d193b1fb8d94284983c445e91bbee6ec9edf82a0b5ae20ce7"}
Sep 15 19:00:03.630993 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:03.630887    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-66b5f6c4f-ldbrg" podStartSLOduration=0.838513401 podStartE2EDuration="1.630862992s" totalImagesPullingTime="792.349591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:00:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:00:03.629949538 +0000 UTC m=+545.117278524" watchObservedRunningTime="2026-09-15 19:00:03.630862992 +0000 UTC m=+545.118191978"
Sep 15 19:00:59.261960 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.261927    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"]
Sep 15 19:00:59.265368 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.265348    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.267418 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.267397    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 19:00:59.267591 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.267576    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-w45l4\""
Sep 15 19:00:59.268296 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.268278    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 15 19:00:59.268429 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.268406    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 19:00:59.275172 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.275149    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"]
Sep 15 19:00:59.328179 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.328145    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4xpl\" (UniqueName: \"kubernetes.io/projected/7a6e67a2-27db-4a2e-8dc8-ab877e76dffb-kube-api-access-g4xpl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf\" (UID: \"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.328179 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.328187    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6e67a2-27db-4a2e-8dc8-ab877e76dffb-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf\" (UID: \"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.328490 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.328228    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6e67a2-27db-4a2e-8dc8-ab877e76dffb-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf\" (UID: \"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.328490 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.328254    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6e67a2-27db-4a2e-8dc8-ab877e76dffb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf\" (UID: \"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.328490 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.328274    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6e67a2-27db-4a2e-8dc8-ab877e76dffb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf\" (UID: \"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.328490 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.328294    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6e67a2-27db-4a2e-8dc8-ab877e76dffb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf\" (UID: \"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.429467 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.429430    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4xpl\" (UniqueName: \"kubernetes.io/projected/7a6e67a2-27db-4a2e-8dc8-ab877e76dffb-kube-api-access-g4xpl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf\" (UID: \"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.429701 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.429481    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6e67a2-27db-4a2e-8dc8-ab877e76dffb-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf\" (UID: \"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.429701 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.429524    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6e67a2-27db-4a2e-8dc8-ab877e76dffb-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf\" (UID: \"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.429701 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.429545    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6e67a2-27db-4a2e-8dc8-ab877e76dffb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf\" (UID: \"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.429701 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.429568    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6e67a2-27db-4a2e-8dc8-ab877e76dffb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf\" (UID: \"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.429701 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.429597    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6e67a2-27db-4a2e-8dc8-ab877e76dffb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf\" (UID: \"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.429969 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.429948    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6e67a2-27db-4a2e-8dc8-ab877e76dffb-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf\" (UID: \"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.430084 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.430060    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6e67a2-27db-4a2e-8dc8-ab877e76dffb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf\" (UID: \"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.430126 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.430104    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6e67a2-27db-4a2e-8dc8-ab877e76dffb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf\" (UID: \"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.431824 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.431804    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6e67a2-27db-4a2e-8dc8-ab877e76dffb-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf\" (UID: \"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.432501 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.432481    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6e67a2-27db-4a2e-8dc8-ab877e76dffb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf\" (UID: \"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.436232 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.436212    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4xpl\" (UniqueName: \"kubernetes.io/projected/7a6e67a2-27db-4a2e-8dc8-ab877e76dffb-kube-api-access-g4xpl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf\" (UID: \"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.578182 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.578088    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:00:59.848043 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.848015    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"]
Sep 15 19:00:59.851448 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:00:59.851417    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6e67a2_27db_4a2e_8dc8_ab877e76dffb.slice/crio-ca3448ac833e8e3662c9905e95bdfd9adcb0354ca1a9dda12a33a8599cf305ff WatchSource:0}: Error finding container ca3448ac833e8e3662c9905e95bdfd9adcb0354ca1a9dda12a33a8599cf305ff: Status 404 returned error can't find the container with id ca3448ac833e8e3662c9905e95bdfd9adcb0354ca1a9dda12a33a8599cf305ff
Sep 15 19:00:59.887169 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:00:59.887125    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf" event={"ID":"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb","Type":"ContainerStarted","Data":"ca3448ac833e8e3662c9905e95bdfd9adcb0354ca1a9dda12a33a8599cf305ff"}
Sep 15 19:01:01.899522 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:01.899484    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf" event={"ID":"7a6e67a2-27db-4a2e-8dc8-ab877e76dffb","Type":"ContainerStarted","Data":"7ba8d095ef7e8878b5bb7354194cdf8804d1bb2f373e61d1be5b2a81c39520f3"}
Sep 15 19:01:09.578790 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:09.578753    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:01:09.581089 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:09.578815    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:01:09.592011 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:09.591986    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:01:09.608298 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:09.608248    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf" podStartSLOduration=9.432240237 podStartE2EDuration="10.608230819s" totalImagesPullingTime="1.175990582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:00:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:01:01.914797679 +0000 UTC m=+603.402126667" watchObservedRunningTime="2026-09-15 19:01:09.608230819 +0000 UTC m=+611.095559831"
Sep 15 19:01:09.949399 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:09.949315    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p5dpf"
Sep 15 19:01:42.237177 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.237143    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"]
Sep 15 19:01:42.241324 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.241304    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.243692 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.243668    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 15 19:01:42.250961 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.250935    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"]
Sep 15 19:01:42.328723 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.328691    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e460485-c0d8-41b2-a4e8-2dc175da4f5e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh\" (UID: \"4e460485-c0d8-41b2-a4e8-2dc175da4f5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.328886 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.328733    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8pkh\" (UniqueName: \"kubernetes.io/projected/4e460485-c0d8-41b2-a4e8-2dc175da4f5e-kube-api-access-g8pkh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh\" (UID: \"4e460485-c0d8-41b2-a4e8-2dc175da4f5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.328886 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.328843    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e460485-c0d8-41b2-a4e8-2dc175da4f5e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh\" (UID: \"4e460485-c0d8-41b2-a4e8-2dc175da4f5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.328886 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.328869    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e460485-c0d8-41b2-a4e8-2dc175da4f5e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh\" (UID: \"4e460485-c0d8-41b2-a4e8-2dc175da4f5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.329017 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.328897    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e460485-c0d8-41b2-a4e8-2dc175da4f5e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh\" (UID: \"4e460485-c0d8-41b2-a4e8-2dc175da4f5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.329017 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.328916    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e460485-c0d8-41b2-a4e8-2dc175da4f5e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh\" (UID: \"4e460485-c0d8-41b2-a4e8-2dc175da4f5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.429875 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.429838    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e460485-c0d8-41b2-a4e8-2dc175da4f5e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh\" (UID: \"4e460485-c0d8-41b2-a4e8-2dc175da4f5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.429875 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.429880    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8pkh\" (UniqueName: \"kubernetes.io/projected/4e460485-c0d8-41b2-a4e8-2dc175da4f5e-kube-api-access-g8pkh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh\" (UID: \"4e460485-c0d8-41b2-a4e8-2dc175da4f5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.430127 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.429948    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e460485-c0d8-41b2-a4e8-2dc175da4f5e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh\" (UID: \"4e460485-c0d8-41b2-a4e8-2dc175da4f5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.430127 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.429969    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e460485-c0d8-41b2-a4e8-2dc175da4f5e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh\" (UID: \"4e460485-c0d8-41b2-a4e8-2dc175da4f5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.430127 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.430102    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e460485-c0d8-41b2-a4e8-2dc175da4f5e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh\" (UID: \"4e460485-c0d8-41b2-a4e8-2dc175da4f5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.430309 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.430151    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e460485-c0d8-41b2-a4e8-2dc175da4f5e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh\" (UID: \"4e460485-c0d8-41b2-a4e8-2dc175da4f5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.430435 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.430415    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e460485-c0d8-41b2-a4e8-2dc175da4f5e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh\" (UID: \"4e460485-c0d8-41b2-a4e8-2dc175da4f5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.430498 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.430443    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e460485-c0d8-41b2-a4e8-2dc175da4f5e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh\" (UID: \"4e460485-c0d8-41b2-a4e8-2dc175da4f5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.430498 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.430473    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e460485-c0d8-41b2-a4e8-2dc175da4f5e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh\" (UID: \"4e460485-c0d8-41b2-a4e8-2dc175da4f5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.432331 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.432304    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e460485-c0d8-41b2-a4e8-2dc175da4f5e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh\" (UID: \"4e460485-c0d8-41b2-a4e8-2dc175da4f5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.432530 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.432513    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e460485-c0d8-41b2-a4e8-2dc175da4f5e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh\" (UID: \"4e460485-c0d8-41b2-a4e8-2dc175da4f5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.439666 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.439634    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8pkh\" (UniqueName: \"kubernetes.io/projected/4e460485-c0d8-41b2-a4e8-2dc175da4f5e-kube-api-access-g8pkh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh\" (UID: \"4e460485-c0d8-41b2-a4e8-2dc175da4f5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.553244 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.553207    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:42.825630 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.825570    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"]
Sep 15 19:01:42.827985 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:01:42.827958    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e460485_c0d8_41b2_a4e8_2dc175da4f5e.slice/crio-816d7a9d3e9db9f4510d51afc80297f8929a76f169683347dec08cd3d1c24ea1 WatchSource:0}: Error finding container 816d7a9d3e9db9f4510d51afc80297f8929a76f169683347dec08cd3d1c24ea1: Status 404 returned error can't find the container with id 816d7a9d3e9db9f4510d51afc80297f8929a76f169683347dec08cd3d1c24ea1
Sep 15 19:01:42.829278 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:42.829263    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:01:43.103320 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:43.103280    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh" event={"ID":"4e460485-c0d8-41b2-a4e8-2dc175da4f5e","Type":"ContainerStarted","Data":"25b3746afb127ada5f9e43afff86f0daef3eb5128f7ef67d7b602208fee19405"}
Sep 15 19:01:43.103486 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:43.103344    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh" event={"ID":"4e460485-c0d8-41b2-a4e8-2dc175da4f5e","Type":"ContainerStarted","Data":"816d7a9d3e9db9f4510d51afc80297f8929a76f169683347dec08cd3d1c24ea1"}
Sep 15 19:01:43.127902 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:43.127848    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh" podStartSLOduration=0.944379304 podStartE2EDuration="1.127828258s" totalImagesPullingTime="183.448954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:01:43.125746867 +0000 UTC m=+644.613075854" watchObservedRunningTime="2026-09-15 19:01:43.127828258 +0000 UTC m=+644.615157242"
Sep 15 19:01:52.553501 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:52.553464    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:52.554063 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:52.553547    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:52.565920 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:52.565889    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:01:53.163426 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:01:53.163342    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5mgzh"
Sep 15 19:03:11.435240 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:11.435150    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66b5f6c4f-ldbrg"]
Sep 15 19:03:11.435814 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:11.435369    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-66b5f6c4f-ldbrg" podUID="25430b78-a108-4ac2-8942-9ac8434b2370" containerName="authorino" containerID="cri-o://fb602fb71caa327701ab9c857c93423effc88aec9c669c489c9aa0a0462e124f" gracePeriod=30
Sep 15 19:03:11.685002 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:11.684975    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66b5f6c4f-ldbrg"
Sep 15 19:03:11.769673 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:11.769635    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25430b78-a108-4ac2-8942-9ac8434b2370-tls-cert\") pod \"25430b78-a108-4ac2-8942-9ac8434b2370\" (UID: \"25430b78-a108-4ac2-8942-9ac8434b2370\") "
Sep 15 19:03:11.769866 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:11.769705    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bk66x\" (UniqueName: \"kubernetes.io/projected/25430b78-a108-4ac2-8942-9ac8434b2370-kube-api-access-bk66x\") pod \"25430b78-a108-4ac2-8942-9ac8434b2370\" (UID: \"25430b78-a108-4ac2-8942-9ac8434b2370\") "
Sep 15 19:03:11.771776 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:11.771750    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25430b78-a108-4ac2-8942-9ac8434b2370-kube-api-access-bk66x" (OuterVolumeSpecName: "kube-api-access-bk66x") pod "25430b78-a108-4ac2-8942-9ac8434b2370" (UID: "25430b78-a108-4ac2-8942-9ac8434b2370"). InnerVolumeSpecName "kube-api-access-bk66x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:03:11.779306 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:11.779285    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25430b78-a108-4ac2-8942-9ac8434b2370-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "25430b78-a108-4ac2-8942-9ac8434b2370" (UID: "25430b78-a108-4ac2-8942-9ac8434b2370"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:03:11.871166 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:11.871114    2555 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25430b78-a108-4ac2-8942-9ac8434b2370-tls-cert\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:03:11.871166 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:11.871156    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk66x\" (UniqueName: \"kubernetes.io/projected/25430b78-a108-4ac2-8942-9ac8434b2370-kube-api-access-bk66x\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:03:12.540824 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:12.540793    2555 generic.go:356] "Generic (PLEG): container finished" podID="25430b78-a108-4ac2-8942-9ac8434b2370" containerID="fb602fb71caa327701ab9c857c93423effc88aec9c669c489c9aa0a0462e124f" exitCode=0
Sep 15 19:03:12.541275 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:12.540835    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66b5f6c4f-ldbrg"
Sep 15 19:03:12.541275 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:12.540876    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66b5f6c4f-ldbrg" event={"ID":"25430b78-a108-4ac2-8942-9ac8434b2370","Type":"ContainerDied","Data":"fb602fb71caa327701ab9c857c93423effc88aec9c669c489c9aa0a0462e124f"}
Sep 15 19:03:12.541275 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:12.540919    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66b5f6c4f-ldbrg" event={"ID":"25430b78-a108-4ac2-8942-9ac8434b2370","Type":"ContainerDied","Data":"d9d78ba6e878452d193b1fb8d94284983c445e91bbee6ec9edf82a0b5ae20ce7"}
Sep 15 19:03:12.541275 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:12.540936    2555 scope.go:117] "RemoveContainer" containerID="fb602fb71caa327701ab9c857c93423effc88aec9c669c489c9aa0a0462e124f"
Sep 15 19:03:12.550024 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:12.550005    2555 scope.go:117] "RemoveContainer" containerID="fb602fb71caa327701ab9c857c93423effc88aec9c669c489c9aa0a0462e124f"
Sep 15 19:03:12.550294 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:03:12.550276    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb602fb71caa327701ab9c857c93423effc88aec9c669c489c9aa0a0462e124f\": container with ID starting with fb602fb71caa327701ab9c857c93423effc88aec9c669c489c9aa0a0462e124f not found: ID does not exist" containerID="fb602fb71caa327701ab9c857c93423effc88aec9c669c489c9aa0a0462e124f"
Sep 15 19:03:12.550365 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:12.550306    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb602fb71caa327701ab9c857c93423effc88aec9c669c489c9aa0a0462e124f"} err="failed to get container status \"fb602fb71caa327701ab9c857c93423effc88aec9c669c489c9aa0a0462e124f\": rpc error: code = NotFound desc = could not find container \"fb602fb71caa327701ab9c857c93423effc88aec9c669c489c9aa0a0462e124f\": container with ID starting with fb602fb71caa327701ab9c857c93423effc88aec9c669c489c9aa0a0462e124f not found: ID does not exist"
Sep 15 19:03:12.563237 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:12.563218    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66b5f6c4f-ldbrg"]
Sep 15 19:03:12.567515 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:12.567495    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-66b5f6c4f-ldbrg"]
Sep 15 19:03:13.040049 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:13.040018    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25430b78-a108-4ac2-8942-9ac8434b2370" path="/var/lib/kubelet/pods/25430b78-a108-4ac2-8942-9ac8434b2370/volumes"
Sep 15 19:03:25.690742 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:25.690710    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6847588947-nwgpc"]
Sep 15 19:03:25.691149 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:25.691082    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25430b78-a108-4ac2-8942-9ac8434b2370" containerName="authorino"
Sep 15 19:03:25.691149 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:25.691094    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="25430b78-a108-4ac2-8942-9ac8434b2370" containerName="authorino"
Sep 15 19:03:25.691222 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:25.691170    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="25430b78-a108-4ac2-8942-9ac8434b2370" containerName="authorino"
Sep 15 19:03:25.693332 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:25.693312    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6847588947-nwgpc"
Sep 15 19:03:25.693957 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:25.693933    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl5rg\" (UniqueName: \"kubernetes.io/projected/8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6-kube-api-access-sl5rg\") pod \"ai-gateway-controller-6847588947-nwgpc\" (UID: \"8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6\") " pod="opendatahub/ai-gateway-controller-6847588947-nwgpc"
Sep 15 19:03:25.695496 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:25.695469    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Sep 15 19:03:25.695634 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:25.695599    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-j4lbd\""
Sep 15 19:03:25.702027 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:25.702004    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6847588947-nwgpc"]
Sep 15 19:03:25.794943 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:25.794904    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl5rg\" (UniqueName: \"kubernetes.io/projected/8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6-kube-api-access-sl5rg\") pod \"ai-gateway-controller-6847588947-nwgpc\" (UID: \"8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6\") " pod="opendatahub/ai-gateway-controller-6847588947-nwgpc"
Sep 15 19:03:25.802936 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:25.802904    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl5rg\" (UniqueName: \"kubernetes.io/projected/8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6-kube-api-access-sl5rg\") pod \"ai-gateway-controller-6847588947-nwgpc\" (UID: \"8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6\") " pod="opendatahub/ai-gateway-controller-6847588947-nwgpc"
Sep 15 19:03:26.004720 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:26.004688    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6847588947-nwgpc"
Sep 15 19:03:26.274113 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:26.274040    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6847588947-nwgpc"]
Sep 15 19:03:26.277386 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:03:26.277350    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c7c4531_1c9c_4624_a7f0_37e3fd8c2ff6.slice/crio-890576d4ce1fbd1ed841917095dd807342ed0b9bee14843b5a65d017cdff468d WatchSource:0}: Error finding container 890576d4ce1fbd1ed841917095dd807342ed0b9bee14843b5a65d017cdff468d: Status 404 returned error can't find the container with id 890576d4ce1fbd1ed841917095dd807342ed0b9bee14843b5a65d017cdff468d
Sep 15 19:03:26.609944 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:26.609853    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6847588947-nwgpc" event={"ID":"8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6","Type":"ContainerStarted","Data":"890576d4ce1fbd1ed841917095dd807342ed0b9bee14843b5a65d017cdff468d"}
Sep 15 19:03:30.632241 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:30.632203    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6847588947-nwgpc" event={"ID":"8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6","Type":"ContainerStarted","Data":"17bfdeb952854eeec93bafaa999598ba1dc4a2305104d10b113e709d37b45fec"}
Sep 15 19:03:30.632657 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:30.632335    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6847588947-nwgpc"
Sep 15 19:03:30.648865 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:30.648820    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6847588947-nwgpc" podStartSLOduration=1.804826191 podStartE2EDuration="5.648803624s" totalImagesPullingTime="3.843977433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:03:30.647809014 +0000 UTC m=+752.135138000" watchObservedRunningTime="2026-09-15 19:03:30.648803624 +0000 UTC m=+752.136132608"
Sep 15 19:03:41.639709 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:41.639676    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6847588947-nwgpc"
Sep 15 19:03:41.841012 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:41.840978    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7f6f96c899-chqcs"]
Sep 15 19:03:41.843603 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:41.843580    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7f6f96c899-chqcs"
Sep 15 19:03:41.850102 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:41.850072    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7f6f96c899-chqcs"]
Sep 15 19:03:41.946332 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:41.946244    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bgfs\" (UniqueName: \"kubernetes.io/projected/a415ba45-bfde-4acc-9544-c1dab8ab93a0-kube-api-access-8bgfs\") pod \"ai-gateway-controller-7f6f96c899-chqcs\" (UID: \"a415ba45-bfde-4acc-9544-c1dab8ab93a0\") " pod="opendatahub/ai-gateway-controller-7f6f96c899-chqcs"
Sep 15 19:03:42.046879 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:42.046845    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bgfs\" (UniqueName: \"kubernetes.io/projected/a415ba45-bfde-4acc-9544-c1dab8ab93a0-kube-api-access-8bgfs\") pod \"ai-gateway-controller-7f6f96c899-chqcs\" (UID: \"a415ba45-bfde-4acc-9544-c1dab8ab93a0\") " pod="opendatahub/ai-gateway-controller-7f6f96c899-chqcs"
Sep 15 19:03:42.054755 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:42.054726    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bgfs\" (UniqueName: \"kubernetes.io/projected/a415ba45-bfde-4acc-9544-c1dab8ab93a0-kube-api-access-8bgfs\") pod \"ai-gateway-controller-7f6f96c899-chqcs\" (UID: \"a415ba45-bfde-4acc-9544-c1dab8ab93a0\") " pod="opendatahub/ai-gateway-controller-7f6f96c899-chqcs"
Sep 15 19:03:42.154954 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:42.154917    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7f6f96c899-chqcs"
Sep 15 19:03:42.420737 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:42.420712    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7f6f96c899-chqcs"]
Sep 15 19:03:42.423359 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:03:42.423328    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda415ba45_bfde_4acc_9544_c1dab8ab93a0.slice/crio-cb96a932f53d47ac61ca445dcf450fdc7484fc66fb990645233f6b3ca7eef534 WatchSource:0}: Error finding container cb96a932f53d47ac61ca445dcf450fdc7484fc66fb990645233f6b3ca7eef534: Status 404 returned error can't find the container with id cb96a932f53d47ac61ca445dcf450fdc7484fc66fb990645233f6b3ca7eef534
Sep 15 19:03:42.691304 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:42.691270    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7f6f96c899-chqcs" event={"ID":"a415ba45-bfde-4acc-9544-c1dab8ab93a0","Type":"ContainerStarted","Data":"cb96a932f53d47ac61ca445dcf450fdc7484fc66fb990645233f6b3ca7eef534"}
Sep 15 19:03:43.697774 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:43.697740    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7f6f96c899-chqcs" event={"ID":"a415ba45-bfde-4acc-9544-c1dab8ab93a0","Type":"ContainerStarted","Data":"38aca01c26cd558b13c0b6abd57b974be5848562cd7bd84384a115ba51385d97"}
Sep 15 19:03:43.698164 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:43.697799    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7f6f96c899-chqcs"
Sep 15 19:03:43.711843 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:43.711790    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7f6f96c899-chqcs" podStartSLOduration=2.167351969 podStartE2EDuration="2.711774307s" totalImagesPullingTime="544.422338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:03:43.710068043 +0000 UTC m=+765.197397019" watchObservedRunningTime="2026-09-15 19:03:43.711774307 +0000 UTC m=+765.199103340"
Sep 15 19:03:54.705032 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:54.704998    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7f6f96c899-chqcs"
Sep 15 19:03:54.745820 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:54.745781    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6847588947-nwgpc"]
Sep 15 19:03:54.746040 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:54.746015    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-6847588947-nwgpc" podUID="8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6" containerName="manager" containerID="cri-o://17bfdeb952854eeec93bafaa999598ba1dc4a2305104d10b113e709d37b45fec" gracePeriod=30
Sep 15 19:03:54.994183 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:54.994154    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6847588947-nwgpc"
Sep 15 19:03:55.079184 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:55.079152    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl5rg\" (UniqueName: \"kubernetes.io/projected/8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6-kube-api-access-sl5rg\") pod \"8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6\" (UID: \"8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6\") "
Sep 15 19:03:55.081315 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:55.081291    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6-kube-api-access-sl5rg" (OuterVolumeSpecName: "kube-api-access-sl5rg") pod "8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6" (UID: "8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6"). InnerVolumeSpecName "kube-api-access-sl5rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:03:55.180284 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:55.180255    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl5rg\" (UniqueName: \"kubernetes.io/projected/8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6-kube-api-access-sl5rg\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:03:55.757991 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:55.757958    2555 generic.go:356] "Generic (PLEG): container finished" podID="8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6" containerID="17bfdeb952854eeec93bafaa999598ba1dc4a2305104d10b113e709d37b45fec" exitCode=0
Sep 15 19:03:55.757991 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:55.757996    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6847588947-nwgpc"
Sep 15 19:03:55.758486 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:55.758043    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6847588947-nwgpc" event={"ID":"8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6","Type":"ContainerDied","Data":"17bfdeb952854eeec93bafaa999598ba1dc4a2305104d10b113e709d37b45fec"}
Sep 15 19:03:55.758486 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:55.758101    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6847588947-nwgpc" event={"ID":"8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6","Type":"ContainerDied","Data":"890576d4ce1fbd1ed841917095dd807342ed0b9bee14843b5a65d017cdff468d"}
Sep 15 19:03:55.758486 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:55.758137    2555 scope.go:117] "RemoveContainer" containerID="17bfdeb952854eeec93bafaa999598ba1dc4a2305104d10b113e709d37b45fec"
Sep 15 19:03:55.766845 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:55.766831    2555 scope.go:117] "RemoveContainer" containerID="17bfdeb952854eeec93bafaa999598ba1dc4a2305104d10b113e709d37b45fec"
Sep 15 19:03:55.767113 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:03:55.767092    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17bfdeb952854eeec93bafaa999598ba1dc4a2305104d10b113e709d37b45fec\": container with ID starting with 17bfdeb952854eeec93bafaa999598ba1dc4a2305104d10b113e709d37b45fec not found: ID does not exist" containerID="17bfdeb952854eeec93bafaa999598ba1dc4a2305104d10b113e709d37b45fec"
Sep 15 19:03:55.767204 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:55.767120    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17bfdeb952854eeec93bafaa999598ba1dc4a2305104d10b113e709d37b45fec"} err="failed to get container status \"17bfdeb952854eeec93bafaa999598ba1dc4a2305104d10b113e709d37b45fec\": rpc error: code = NotFound desc = could not find container \"17bfdeb952854eeec93bafaa999598ba1dc4a2305104d10b113e709d37b45fec\": container with ID starting with 17bfdeb952854eeec93bafaa999598ba1dc4a2305104d10b113e709d37b45fec not found: ID does not exist"
Sep 15 19:03:55.780687 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:55.780661    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6847588947-nwgpc"]
Sep 15 19:03:55.784039 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:55.784018    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-6847588947-nwgpc"]
Sep 15 19:03:57.040093 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:03:57.040065    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6" path="/var/lib/kubelet/pods/8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6/volumes"
Sep 15 19:04:16.204131 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:16.204098    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8936-6c9ec7"]
Sep 15 19:04:16.204684 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:16.204499    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6" containerName="manager"
Sep 15 19:04:16.204684 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:16.204511    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6" containerName="manager"
Sep 15 19:04:16.204684 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:16.204588    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c7c4531-1c9c-4624-a7f0-37e3fd8c2ff6" containerName="manager"
Sep 15 19:04:16.211652 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:16.211630    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-6c9ec7"
Sep 15 19:04:16.213399 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:16.213371    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8936-6c9ec7"]
Sep 15 19:04:16.215864 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:16.215843    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zrrnn\""
Sep 15 19:04:16.271541 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:16.271508    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q452j\" (UniqueName: \"kubernetes.io/projected/de25bfbb-46ee-44f0-a589-5fc8b73b53fe-kube-api-access-q452j\") pod \"test-curl-8936-6c9ec7\" (UID: \"de25bfbb-46ee-44f0-a589-5fc8b73b53fe\") " pod="opendatahub/test-curl-8936-6c9ec7"
Sep 15 19:04:16.372290 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:16.372250    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q452j\" (UniqueName: \"kubernetes.io/projected/de25bfbb-46ee-44f0-a589-5fc8b73b53fe-kube-api-access-q452j\") pod \"test-curl-8936-6c9ec7\" (UID: \"de25bfbb-46ee-44f0-a589-5fc8b73b53fe\") " pod="opendatahub/test-curl-8936-6c9ec7"
Sep 15 19:04:16.379886 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:16.379853    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q452j\" (UniqueName: \"kubernetes.io/projected/de25bfbb-46ee-44f0-a589-5fc8b73b53fe-kube-api-access-q452j\") pod \"test-curl-8936-6c9ec7\" (UID: \"de25bfbb-46ee-44f0-a589-5fc8b73b53fe\") " pod="opendatahub/test-curl-8936-6c9ec7"
Sep 15 19:04:16.521169 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:16.521132    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-6c9ec7"
Sep 15 19:04:16.788945 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:16.788920    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8936-6c9ec7"]
Sep 15 19:04:16.791150 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:04:16.791120    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde25bfbb_46ee_44f0_a589_5fc8b73b53fe.slice/crio-bcae9c474b3c251f78c17626dd0bc0765e7ddefd3d9af81203cecf19fd638673 WatchSource:0}: Error finding container bcae9c474b3c251f78c17626dd0bc0765e7ddefd3d9af81203cecf19fd638673: Status 404 returned error can't find the container with id bcae9c474b3c251f78c17626dd0bc0765e7ddefd3d9af81203cecf19fd638673
Sep 15 19:04:16.862924 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:16.862888    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8936-6c9ec7" event={"ID":"de25bfbb-46ee-44f0-a589-5fc8b73b53fe","Type":"ContainerStarted","Data":"bcae9c474b3c251f78c17626dd0bc0765e7ddefd3d9af81203cecf19fd638673"}
Sep 15 19:04:18.866420 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:18.866249    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"]
Sep 15 19:04:18.879398 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:18.879257    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"
Sep 15 19:04:18.881534 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:18.881503    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"]
Sep 15 19:04:18.882004 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:18.881879    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-7c96ca-serving-cert\""
Sep 15 19:04:18.882249 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:18.882227    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:04:18.883064 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:18.882809    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5ksbj\""
Sep 15 19:04:18.883064 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:18.882866    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 19:04:18.883064 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:18.883042    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 19:04:18.998716 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:18.998687    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5efbd9d-fd2d-473d-921d-7adad6724f90-metrics-certs\") pod \"maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb\" (UID: \"c5efbd9d-fd2d-473d-921d-7adad6724f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"
Sep 15 19:04:18.998834 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:18.998725    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kth88\" (UniqueName: \"kubernetes.io/projected/c5efbd9d-fd2d-473d-921d-7adad6724f90-kube-api-access-kth88\") pod \"maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb\" (UID: \"c5efbd9d-fd2d-473d-921d-7adad6724f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"
Sep 15 19:04:18.998834 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:18.998798    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5efbd9d-fd2d-473d-921d-7adad6724f90-maas-api-tls\") pod \"maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb\" (UID: \"c5efbd9d-fd2d-473d-921d-7adad6724f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"
Sep 15 19:04:18.998961 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:18.998872    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5efbd9d-fd2d-473d-921d-7adad6724f90-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb\" (UID: \"c5efbd9d-fd2d-473d-921d-7adad6724f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"
Sep 15 19:04:19.100282 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:19.100242    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5efbd9d-fd2d-473d-921d-7adad6724f90-metrics-certs\") pod \"maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb\" (UID: \"c5efbd9d-fd2d-473d-921d-7adad6724f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"
Sep 15 19:04:19.100282 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:19.100281    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kth88\" (UniqueName: \"kubernetes.io/projected/c5efbd9d-fd2d-473d-921d-7adad6724f90-kube-api-access-kth88\") pod \"maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb\" (UID: \"c5efbd9d-fd2d-473d-921d-7adad6724f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"
Sep 15 19:04:19.100539 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:19.100334    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5efbd9d-fd2d-473d-921d-7adad6724f90-maas-api-tls\") pod \"maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb\" (UID: \"c5efbd9d-fd2d-473d-921d-7adad6724f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"
Sep 15 19:04:19.100539 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:19.100397    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5efbd9d-fd2d-473d-921d-7adad6724f90-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb\" (UID: \"c5efbd9d-fd2d-473d-921d-7adad6724f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"
Sep 15 19:04:19.100539 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:04:19.100494    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-serving-cert: secret "maas-api-e2e-worker-w3-7c96ca-serving-cert" not found
Sep 15 19:04:19.100749 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:04:19.100573    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5efbd9d-fd2d-473d-921d-7adad6724f90-maas-api-tls podName:c5efbd9d-fd2d-473d-921d-7adad6724f90 nodeName:}" failed. No retries permitted until 2026-09-15 19:04:19.600552652 +0000 UTC m=+801.087881620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c5efbd9d-fd2d-473d-921d-7adad6724f90-maas-api-tls") pod "maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb" (UID: "c5efbd9d-fd2d-473d-921d-7adad6724f90") : secret "maas-api-e2e-worker-w3-7c96ca-serving-cert" not found
Sep 15 19:04:19.101094 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:19.101070    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5efbd9d-fd2d-473d-921d-7adad6724f90-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb\" (UID: \"c5efbd9d-fd2d-473d-921d-7adad6724f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"
Sep 15 19:04:19.102825 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:19.102804    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5efbd9d-fd2d-473d-921d-7adad6724f90-metrics-certs\") pod \"maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb\" (UID: \"c5efbd9d-fd2d-473d-921d-7adad6724f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"
Sep 15 19:04:19.109029 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:19.109000    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kth88\" (UniqueName: \"kubernetes.io/projected/c5efbd9d-fd2d-473d-921d-7adad6724f90-kube-api-access-kth88\") pod \"maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb\" (UID: \"c5efbd9d-fd2d-473d-921d-7adad6724f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"
Sep 15 19:04:19.605933 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:19.605893    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5efbd9d-fd2d-473d-921d-7adad6724f90-maas-api-tls\") pod \"maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb\" (UID: \"c5efbd9d-fd2d-473d-921d-7adad6724f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"
Sep 15 19:04:19.608246 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:19.608219    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5efbd9d-fd2d-473d-921d-7adad6724f90-maas-api-tls\") pod \"maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb\" (UID: \"c5efbd9d-fd2d-473d-921d-7adad6724f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"
Sep 15 19:04:19.792281 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:19.792245    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"
Sep 15 19:04:19.884590 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:19.884499    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8936-6c9ec7" event={"ID":"de25bfbb-46ee-44f0-a589-5fc8b73b53fe","Type":"ContainerStarted","Data":"75011431177c2aba685cd6761bd64c043ea36ff6e46dd3663c423ba20fa8d251"}
Sep 15 19:04:19.897758 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:19.897702    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8936-6c9ec7" podStartSLOduration=1.708241182 podStartE2EDuration="3.897684534s" totalImagesPullingTime="2.189443352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:04:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:04:19.897171897 +0000 UTC m=+801.384500883" watchObservedRunningTime="2026-09-15 19:04:19.897684534 +0000 UTC m=+801.385013523"
Sep 15 19:04:20.071096 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.071064    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"]
Sep 15 19:04:20.072475 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:04:20.072448    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5efbd9d_fd2d_473d_921d_7adad6724f90.slice/crio-88188defa8aa6ffbed0f6340ef380bc9def6d14d02d33d527d0e74cb4061bb04 WatchSource:0}: Error finding container 88188defa8aa6ffbed0f6340ef380bc9def6d14d02d33d527d0e74cb4061bb04: Status 404 returned error can't find the container with id 88188defa8aa6ffbed0f6340ef380bc9def6d14d02d33d527d0e74cb4061bb04
Sep 15 19:04:20.508417 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.508376    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8936-6c9ec7"]
Sep 15 19:04:20.522017 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.521989    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8936-6c9ec7"]
Sep 15 19:04:20.642700 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.642624    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8936-521700"]
Sep 15 19:04:20.643149 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.643125    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de25bfbb-46ee-44f0-a589-5fc8b73b53fe" containerName="test-curl-8936-6c9ec7"
Sep 15 19:04:20.643149 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.643147    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="de25bfbb-46ee-44f0-a589-5fc8b73b53fe" containerName="test-curl-8936-6c9ec7"
Sep 15 19:04:20.643343 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.643239    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="de25bfbb-46ee-44f0-a589-5fc8b73b53fe" containerName="test-curl-8936-6c9ec7"
Sep 15 19:04:20.646673 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.646647    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-521700"
Sep 15 19:04:20.655797 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.655634    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8936-521700"]
Sep 15 19:04:20.715776 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.715744    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxrtk\" (UniqueName: \"kubernetes.io/projected/d039cf04-01d8-44fb-9555-99eaa4e68044-kube-api-access-mxrtk\") pod \"test-curl-8936-521700\" (UID: \"d039cf04-01d8-44fb-9555-99eaa4e68044\") " pod="opendatahub/test-curl-8936-521700"
Sep 15 19:04:20.775149 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.774952    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"]
Sep 15 19:04:20.779044 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.778987    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"
Sep 15 19:04:20.781353 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.781317    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d5559a-serving-cert\""
Sep 15 19:04:20.788509 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.788483    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"]
Sep 15 19:04:20.817324 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.817291    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrtl8\" (UniqueName: \"kubernetes.io/projected/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-kube-api-access-jrtl8\") pod \"maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr\" (UID: \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"
Sep 15 19:04:20.817474 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.817355    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr\" (UID: \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"
Sep 15 19:04:20.817474 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.817381    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-metrics-certs\") pod \"maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr\" (UID: \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"
Sep 15 19:04:20.817474 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.817454    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-maas-api-tls\") pod \"maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr\" (UID: \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"
Sep 15 19:04:20.817680 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.817561    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxrtk\" (UniqueName: \"kubernetes.io/projected/d039cf04-01d8-44fb-9555-99eaa4e68044-kube-api-access-mxrtk\") pod \"test-curl-8936-521700\" (UID: \"d039cf04-01d8-44fb-9555-99eaa4e68044\") " pod="opendatahub/test-curl-8936-521700"
Sep 15 19:04:20.825678 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.825646    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxrtk\" (UniqueName: \"kubernetes.io/projected/d039cf04-01d8-44fb-9555-99eaa4e68044-kube-api-access-mxrtk\") pod \"test-curl-8936-521700\" (UID: \"d039cf04-01d8-44fb-9555-99eaa4e68044\") " pod="opendatahub/test-curl-8936-521700"
Sep 15 19:04:20.892676 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.892635    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb" event={"ID":"c5efbd9d-fd2d-473d-921d-7adad6724f90","Type":"ContainerStarted","Data":"88188defa8aa6ffbed0f6340ef380bc9def6d14d02d33d527d0e74cb4061bb04"}
Sep 15 19:04:20.893181 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:04:20.892863    2555 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8936-6c9ec7" container="test-curl-8936-6c9ec7"
Sep 15 19:04:20.918897 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.918860    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr\" (UID: \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"
Sep 15 19:04:20.919094 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.918909    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-metrics-certs\") pod \"maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr\" (UID: \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"
Sep 15 19:04:20.919094 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.918964    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-maas-api-tls\") pod \"maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr\" (UID: \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"
Sep 15 19:04:20.919220 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.919098    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrtl8\" (UniqueName: \"kubernetes.io/projected/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-kube-api-access-jrtl8\") pod \"maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr\" (UID: \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"
Sep 15 19:04:20.919280 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:04:20.919213    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-serving-cert: secret "maas-api-e2e-inf-a-d5559a-serving-cert" not found
Sep 15 19:04:20.919331 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:04:20.919289    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-maas-api-tls podName:cc25a84c-4256-44f4-8f51-3cd1dc33cf80 nodeName:}" failed. No retries permitted until 2026-09-15 19:04:21.419268037 +0000 UTC m=+802.906597001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-maas-api-tls") pod "maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr" (UID: "cc25a84c-4256-44f4-8f51-3cd1dc33cf80") : secret "maas-api-e2e-inf-a-d5559a-serving-cert" not found
Sep 15 19:04:20.919857 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.919829    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr\" (UID: \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"
Sep 15 19:04:20.922500 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.922466    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-metrics-certs\") pod \"maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr\" (UID: \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"
Sep 15 19:04:20.928477 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.928426    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrtl8\" (UniqueName: \"kubernetes.io/projected/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-kube-api-access-jrtl8\") pod \"maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr\" (UID: \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"
Sep 15 19:04:20.962553 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:20.962517    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-521700"
Sep 15 19:04:21.253073 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:21.253042    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8936-521700"]
Sep 15 19:04:21.256031 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:04:21.256001    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd039cf04_01d8_44fb_9555_99eaa4e68044.slice/crio-48c1a7caded088007065462e22b7d8fe27b28ba324008f6ac6cd8b1c528e3804 WatchSource:0}: Error finding container 48c1a7caded088007065462e22b7d8fe27b28ba324008f6ac6cd8b1c528e3804: Status 404 returned error can't find the container with id 48c1a7caded088007065462e22b7d8fe27b28ba324008f6ac6cd8b1c528e3804
Sep 15 19:04:21.426222 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:21.426036    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-maas-api-tls\") pod \"maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr\" (UID: \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"
Sep 15 19:04:21.428808 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:21.428781    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-maas-api-tls\") pod \"maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr\" (UID: \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"
Sep 15 19:04:21.694102 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:21.694030    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"
Sep 15 19:04:21.899966 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:21.899927    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8936-521700" event={"ID":"d039cf04-01d8-44fb-9555-99eaa4e68044","Type":"ContainerStarted","Data":"6e2a5a1a230fa020af16811a03b9149a058c2df949eaef5e45eb956d33c0cb6c"}
Sep 15 19:04:21.900426 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:21.899979    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8936-521700" event={"ID":"d039cf04-01d8-44fb-9555-99eaa4e68044","Type":"ContainerStarted","Data":"48c1a7caded088007065462e22b7d8fe27b28ba324008f6ac6cd8b1c528e3804"}
Sep 15 19:04:21.900426 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:21.900026    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8936-6c9ec7" podUID="de25bfbb-46ee-44f0-a589-5fc8b73b53fe" containerName="test-curl-8936-6c9ec7" containerID="cri-o://75011431177c2aba685cd6761bd64c043ea36ff6e46dd3663c423ba20fa8d251" gracePeriod=2
Sep 15 19:04:21.913376 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:21.913317    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8936-521700" podStartSLOduration=1.91329547 podStartE2EDuration="1.91329547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:04:21.912915734 +0000 UTC m=+803.400244714" watchObservedRunningTime="2026-09-15 19:04:21.91329547 +0000 UTC m=+803.400624455"
Sep 15 19:04:22.601286 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:22.601220    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8936-521700"]
Sep 15 19:04:22.607662 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:22.607634    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8936-521700"]
Sep 15 19:04:22.731315 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:22.731246    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"]
Sep 15 19:04:22.733445 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:04:22.733410    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc25a84c_4256_44f4_8f51_3cd1dc33cf80.slice/crio-71a246c24f9e00a84d587d601d1947cad8cd4dea84119b0487d4f6c1a05b821c WatchSource:0}: Error finding container 71a246c24f9e00a84d587d601d1947cad8cd4dea84119b0487d4f6c1a05b821c: Status 404 returned error can't find the container with id 71a246c24f9e00a84d587d601d1947cad8cd4dea84119b0487d4f6c1a05b821c
Sep 15 19:04:22.906728 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:22.906650    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr" event={"ID":"cc25a84c-4256-44f4-8f51-3cd1dc33cf80","Type":"ContainerStarted","Data":"71a246c24f9e00a84d587d601d1947cad8cd4dea84119b0487d4f6c1a05b821c"}
Sep 15 19:04:22.908174 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:22.908140    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb" event={"ID":"c5efbd9d-fd2d-473d-921d-7adad6724f90","Type":"ContainerStarted","Data":"079fba945a6dc6ba204fe439585aabb7e156fb16475b9a4cf40ffc37227b1b67"}
Sep 15 19:04:22.908394 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:04:22.908375    2555 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8936-521700" container="test-curl-8936-521700"
Sep 15 19:04:22.931677 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:22.931624    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb" podStartSLOduration=2.486561631 podStartE2EDuration="4.931591845s" totalImagesPullingTime="2.445030214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:04:22.92989518 +0000 UTC m=+804.417224167" watchObservedRunningTime="2026-09-15 19:04:22.931591845 +0000 UTC m=+804.418920829"
Sep 15 19:04:23.916643 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:23.916597    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr" event={"ID":"cc25a84c-4256-44f4-8f51-3cd1dc33cf80","Type":"ContainerStarted","Data":"e2cb69da4e26a2e5e1255f642387fd6cb00e7de007bfe588355b9a335e745ae6"}
Sep 15 19:04:23.917001 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:23.916796    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8936-521700" podUID="d039cf04-01d8-44fb-9555-99eaa4e68044" containerName="test-curl-8936-521700" containerID="cri-o://6e2a5a1a230fa020af16811a03b9149a058c2df949eaef5e45eb956d33c0cb6c" gracePeriod=2
Sep 15 19:04:23.917001 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:23.916973    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"
Sep 15 19:04:23.917113 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:23.917035    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"
Sep 15 19:04:23.932989 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:23.932937    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr" podStartSLOduration=3.590855561 podStartE2EDuration="3.932923032s" totalImagesPullingTime="342.067471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:04:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:04:23.931420934 +0000 UTC m=+805.418749922" watchObservedRunningTime="2026-09-15 19:04:23.932923032 +0000 UTC m=+805.420252017"
Sep 15 19:04:24.090532 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:24.090506    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-6c9ec7"
Sep 15 19:04:24.153400 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:24.153348    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q452j\" (UniqueName: \"kubernetes.io/projected/de25bfbb-46ee-44f0-a589-5fc8b73b53fe-kube-api-access-q452j\") pod \"de25bfbb-46ee-44f0-a589-5fc8b73b53fe\" (UID: \"de25bfbb-46ee-44f0-a589-5fc8b73b53fe\") "
Sep 15 19:04:24.155562 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:24.155488    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de25bfbb-46ee-44f0-a589-5fc8b73b53fe-kube-api-access-q452j" (OuterVolumeSpecName: "kube-api-access-q452j") pod "de25bfbb-46ee-44f0-a589-5fc8b73b53fe" (UID: "de25bfbb-46ee-44f0-a589-5fc8b73b53fe"). InnerVolumeSpecName "kube-api-access-q452j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:04:24.254597 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:24.254551    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q452j\" (UniqueName: \"kubernetes.io/projected/de25bfbb-46ee-44f0-a589-5fc8b73b53fe-kube-api-access-q452j\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:04:24.922812 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:24.922784    2555 generic.go:356] "Generic (PLEG): container finished" podID="de25bfbb-46ee-44f0-a589-5fc8b73b53fe" containerID="75011431177c2aba685cd6761bd64c043ea36ff6e46dd3663c423ba20fa8d251" exitCode=137
Sep 15 19:04:24.923203 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:24.922828    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-6c9ec7"
Sep 15 19:04:24.923203 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:24.922900    2555 scope.go:117] "RemoveContainer" containerID="75011431177c2aba685cd6761bd64c043ea36ff6e46dd3663c423ba20fa8d251"
Sep 15 19:04:24.931817 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:24.931798    2555 scope.go:117] "RemoveContainer" containerID="75011431177c2aba685cd6761bd64c043ea36ff6e46dd3663c423ba20fa8d251"
Sep 15 19:04:24.932075 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:04:24.932059    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75011431177c2aba685cd6761bd64c043ea36ff6e46dd3663c423ba20fa8d251\": container with ID starting with 75011431177c2aba685cd6761bd64c043ea36ff6e46dd3663c423ba20fa8d251 not found: ID does not exist" containerID="75011431177c2aba685cd6761bd64c043ea36ff6e46dd3663c423ba20fa8d251"
Sep 15 19:04:24.932201 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:24.932081    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75011431177c2aba685cd6761bd64c043ea36ff6e46dd3663c423ba20fa8d251"} err="failed to get container status \"75011431177c2aba685cd6761bd64c043ea36ff6e46dd3663c423ba20fa8d251\": rpc error: code = NotFound desc = could not find container \"75011431177c2aba685cd6761bd64c043ea36ff6e46dd3663c423ba20fa8d251\": container with ID starting with 75011431177c2aba685cd6761bd64c043ea36ff6e46dd3663c423ba20fa8d251 not found: ID does not exist"
Sep 15 19:04:24.932296 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:04:24.932281    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8936-6c9ec7\" is forbidden: User \"system:node:ip-10-0-132-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-28.ec2.internal' and this object" podUID="de25bfbb-46ee-44f0-a589-5fc8b73b53fe" pod="opendatahub/test-curl-8936-6c9ec7"
Sep 15 19:04:25.040797 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:25.040768    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de25bfbb-46ee-44f0-a589-5fc8b73b53fe" path="/var/lib/kubelet/pods/de25bfbb-46ee-44f0-a589-5fc8b73b53fe/volumes"
Sep 15 19:04:26.104597 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:26.104574    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-521700"
Sep 15 19:04:26.175017 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:26.174936    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxrtk\" (UniqueName: \"kubernetes.io/projected/d039cf04-01d8-44fb-9555-99eaa4e68044-kube-api-access-mxrtk\") pod \"d039cf04-01d8-44fb-9555-99eaa4e68044\" (UID: \"d039cf04-01d8-44fb-9555-99eaa4e68044\") "
Sep 15 19:04:26.177006 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:26.176978    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d039cf04-01d8-44fb-9555-99eaa4e68044-kube-api-access-mxrtk" (OuterVolumeSpecName: "kube-api-access-mxrtk") pod "d039cf04-01d8-44fb-9555-99eaa4e68044" (UID: "d039cf04-01d8-44fb-9555-99eaa4e68044"). InnerVolumeSpecName "kube-api-access-mxrtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:04:26.276680 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:26.276648    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxrtk\" (UniqueName: \"kubernetes.io/projected/d039cf04-01d8-44fb-9555-99eaa4e68044-kube-api-access-mxrtk\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:04:26.936303 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:26.936263    2555 generic.go:356] "Generic (PLEG): container finished" podID="d039cf04-01d8-44fb-9555-99eaa4e68044" containerID="6e2a5a1a230fa020af16811a03b9149a058c2df949eaef5e45eb956d33c0cb6c" exitCode=137
Sep 15 19:04:26.936515 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:26.936315    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8936-521700"
Sep 15 19:04:26.936515 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:26.936381    2555 scope.go:117] "RemoveContainer" containerID="6e2a5a1a230fa020af16811a03b9149a058c2df949eaef5e45eb956d33c0cb6c"
Sep 15 19:04:26.949192 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:26.948971    2555 scope.go:117] "RemoveContainer" containerID="6e2a5a1a230fa020af16811a03b9149a058c2df949eaef5e45eb956d33c0cb6c"
Sep 15 19:04:26.949334 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:04:26.949303    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e2a5a1a230fa020af16811a03b9149a058c2df949eaef5e45eb956d33c0cb6c\": container with ID starting with 6e2a5a1a230fa020af16811a03b9149a058c2df949eaef5e45eb956d33c0cb6c not found: ID does not exist" containerID="6e2a5a1a230fa020af16811a03b9149a058c2df949eaef5e45eb956d33c0cb6c"
Sep 15 19:04:26.949404 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:26.949352    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e2a5a1a230fa020af16811a03b9149a058c2df949eaef5e45eb956d33c0cb6c"} err="failed to get container status \"6e2a5a1a230fa020af16811a03b9149a058c2df949eaef5e45eb956d33c0cb6c\": rpc error: code = NotFound desc = could not find container \"6e2a5a1a230fa020af16811a03b9149a058c2df949eaef5e45eb956d33c0cb6c\": container with ID starting with 6e2a5a1a230fa020af16811a03b9149a058c2df949eaef5e45eb956d33c0cb6c not found: ID does not exist"
Sep 15 19:04:26.950866 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:04:26.950839    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8936-521700\" is forbidden: User \"system:node:ip-10-0-132-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-28.ec2.internal' and this object" podUID="d039cf04-01d8-44fb-9555-99eaa4e68044" pod="opendatahub/test-curl-8936-521700"
Sep 15 19:04:27.040101 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:27.040069    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d039cf04-01d8-44fb-9555-99eaa4e68044" path="/var/lib/kubelet/pods/d039cf04-01d8-44fb-9555-99eaa4e68044/volumes"
Sep 15 19:04:28.672381 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:28.672350    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"]
Sep 15 19:04:28.672792 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:28.672778    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d039cf04-01d8-44fb-9555-99eaa4e68044" containerName="test-curl-8936-521700"
Sep 15 19:04:28.672792 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:28.672793    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d039cf04-01d8-44fb-9555-99eaa4e68044" containerName="test-curl-8936-521700"
Sep 15 19:04:28.672884 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:28.672855    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d039cf04-01d8-44fb-9555-99eaa4e68044" containerName="test-curl-8936-521700"
Sep 15 19:04:28.677546 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:28.677524    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"
Sep 15 19:04:28.679466 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:28.679442    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4df819e5-serving-cert\""
Sep 15 19:04:28.688462 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:28.688433    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"]
Sep 15 19:04:28.800710 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:28.800673    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h565\" (UniqueName: \"kubernetes.io/projected/50d30e09-6ad4-4750-b08b-23bb45ffd12f-kube-api-access-5h565\") pod \"maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq\" (UID: \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"
Sep 15 19:04:28.800710 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:28.800713    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d30e09-6ad4-4750-b08b-23bb45ffd12f-openshift-service-ca\") pod \"maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq\" (UID: \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"
Sep 15 19:04:28.800969 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:28.800749    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50d30e09-6ad4-4750-b08b-23bb45ffd12f-metrics-certs\") pod \"maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq\" (UID: \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"
Sep 15 19:04:28.800969 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:28.800859    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50d30e09-6ad4-4750-b08b-23bb45ffd12f-maas-api-tls\") pod \"maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq\" (UID: \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"
Sep 15 19:04:28.902417 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:28.902379    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50d30e09-6ad4-4750-b08b-23bb45ffd12f-maas-api-tls\") pod \"maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq\" (UID: \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"
Sep 15 19:04:28.902584 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:28.902460    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h565\" (UniqueName: \"kubernetes.io/projected/50d30e09-6ad4-4750-b08b-23bb45ffd12f-kube-api-access-5h565\") pod \"maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq\" (UID: \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"
Sep 15 19:04:28.902584 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:28.902479    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d30e09-6ad4-4750-b08b-23bb45ffd12f-openshift-service-ca\") pod \"maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq\" (UID: \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"
Sep 15 19:04:28.902584 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:28.902505    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50d30e09-6ad4-4750-b08b-23bb45ffd12f-metrics-certs\") pod \"maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq\" (UID: \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"
Sep 15 19:04:28.902584 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:04:28.902537    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-serving-cert: secret "maas-api-e2e-ait-4df819e5-serving-cert" not found
Sep 15 19:04:28.902784 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:04:28.902642    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50d30e09-6ad4-4750-b08b-23bb45ffd12f-maas-api-tls podName:50d30e09-6ad4-4750-b08b-23bb45ffd12f nodeName:}" failed. No retries permitted until 2026-09-15 19:04:29.402598623 +0000 UTC m=+810.889927593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/50d30e09-6ad4-4750-b08b-23bb45ffd12f-maas-api-tls") pod "maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq" (UID: "50d30e09-6ad4-4750-b08b-23bb45ffd12f") : secret "maas-api-e2e-ait-4df819e5-serving-cert" not found
Sep 15 19:04:28.903224 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:28.903195    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d30e09-6ad4-4750-b08b-23bb45ffd12f-openshift-service-ca\") pod \"maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq\" (UID: \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"
Sep 15 19:04:28.905002 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:28.904972    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50d30e09-6ad4-4750-b08b-23bb45ffd12f-metrics-certs\") pod \"maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq\" (UID: \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"
Sep 15 19:04:28.910787 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:28.910767    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h565\" (UniqueName: \"kubernetes.io/projected/50d30e09-6ad4-4750-b08b-23bb45ffd12f-kube-api-access-5h565\") pod \"maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq\" (UID: \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"
Sep 15 19:04:29.407449 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:29.407415    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50d30e09-6ad4-4750-b08b-23bb45ffd12f-maas-api-tls\") pod \"maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq\" (UID: \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"
Sep 15 19:04:29.410075 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:29.410050    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50d30e09-6ad4-4750-b08b-23bb45ffd12f-maas-api-tls\") pod \"maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq\" (UID: \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"
Sep 15 19:04:29.589658 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:29.589619    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"
Sep 15 19:04:29.858795 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:29.858770    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"]
Sep 15 19:04:29.861673 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:04:29.861640    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d30e09_6ad4_4750_b08b_23bb45ffd12f.slice/crio-2d9cddfd4edcd9969fba90dac5c6fc4dd0f337e82362089c8dd48efd274f7287 WatchSource:0}: Error finding container 2d9cddfd4edcd9969fba90dac5c6fc4dd0f337e82362089c8dd48efd274f7287: Status 404 returned error can't find the container with id 2d9cddfd4edcd9969fba90dac5c6fc4dd0f337e82362089c8dd48efd274f7287
Sep 15 19:04:29.928776 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:29.928749    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"
Sep 15 19:04:29.929259 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:29.929236    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"
Sep 15 19:04:29.954537 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:29.954503    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq" event={"ID":"50d30e09-6ad4-4750-b08b-23bb45ffd12f","Type":"ContainerStarted","Data":"2d9cddfd4edcd9969fba90dac5c6fc4dd0f337e82362089c8dd48efd274f7287"}
Sep 15 19:04:32.971062 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:32.971022    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq" event={"ID":"50d30e09-6ad4-4750-b08b-23bb45ffd12f","Type":"ContainerStarted","Data":"bebbfeb30928715e06c80602c7d4dc9fc1e12c34f7acb785b5a674ab070a860f"}
Sep 15 19:04:32.971473 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:32.971173    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"
Sep 15 19:04:32.990031 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:32.989978    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq" podStartSLOduration=2.567832084 podStartE2EDuration="4.989961631s" totalImagesPullingTime="2.422129547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:04:32.987287925 +0000 UTC m=+814.474616910" watchObservedRunningTime="2026-09-15 19:04:32.989961631 +0000 UTC m=+814.477290615"
Sep 15 19:04:38.981864 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:38.981831    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"
Sep 15 19:04:43.631151 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:43.631116    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"]
Sep 15 19:04:43.642034 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:43.642011    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"
Sep 15 19:04:43.644314 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:43.644291    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-08e9c4-serving-cert\""
Sep 15 19:04:43.646017 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:43.645985    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"]
Sep 15 19:04:43.647157 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:43.647134    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqk99\" (UniqueName: \"kubernetes.io/projected/e34a51b1-98b4-49a7-914b-ab0949e57525-kube-api-access-rqk99\") pod \"maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl\" (UID: \"e34a51b1-98b4-49a7-914b-ab0949e57525\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"
Sep 15 19:04:43.647551 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:43.647184    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e34a51b1-98b4-49a7-914b-ab0949e57525-metrics-certs\") pod \"maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl\" (UID: \"e34a51b1-98b4-49a7-914b-ab0949e57525\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"
Sep 15 19:04:43.647551 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:43.647283    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e34a51b1-98b4-49a7-914b-ab0949e57525-openshift-service-ca\") pod \"maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl\" (UID: \"e34a51b1-98b4-49a7-914b-ab0949e57525\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"
Sep 15 19:04:43.647551 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:43.647323    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e34a51b1-98b4-49a7-914b-ab0949e57525-maas-api-tls\") pod \"maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl\" (UID: \"e34a51b1-98b4-49a7-914b-ab0949e57525\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"
Sep 15 19:04:43.748364 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:43.748322    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e34a51b1-98b4-49a7-914b-ab0949e57525-maas-api-tls\") pod \"maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl\" (UID: \"e34a51b1-98b4-49a7-914b-ab0949e57525\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"
Sep 15 19:04:43.748579 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:43.748422    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqk99\" (UniqueName: \"kubernetes.io/projected/e34a51b1-98b4-49a7-914b-ab0949e57525-kube-api-access-rqk99\") pod \"maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl\" (UID: \"e34a51b1-98b4-49a7-914b-ab0949e57525\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"
Sep 15 19:04:43.748579 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:43.748455    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e34a51b1-98b4-49a7-914b-ab0949e57525-metrics-certs\") pod \"maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl\" (UID: \"e34a51b1-98b4-49a7-914b-ab0949e57525\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"
Sep 15 19:04:43.748579 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:04:43.748483    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-serving-cert: secret "maas-api-e2e-inf-b-08e9c4-serving-cert" not found
Sep 15 19:04:43.748579 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:04:43.748566    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e34a51b1-98b4-49a7-914b-ab0949e57525-maas-api-tls podName:e34a51b1-98b4-49a7-914b-ab0949e57525 nodeName:}" failed. No retries permitted until 2026-09-15 19:04:44.24854325 +0000 UTC m=+825.735872230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e34a51b1-98b4-49a7-914b-ab0949e57525-maas-api-tls") pod "maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl" (UID: "e34a51b1-98b4-49a7-914b-ab0949e57525") : secret "maas-api-e2e-inf-b-08e9c4-serving-cert" not found
Sep 15 19:04:43.748814 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:43.748494    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e34a51b1-98b4-49a7-914b-ab0949e57525-openshift-service-ca\") pod \"maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl\" (UID: \"e34a51b1-98b4-49a7-914b-ab0949e57525\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"
Sep 15 19:04:43.749192 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:43.749167    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e34a51b1-98b4-49a7-914b-ab0949e57525-openshift-service-ca\") pod \"maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl\" (UID: \"e34a51b1-98b4-49a7-914b-ab0949e57525\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"
Sep 15 19:04:43.751040 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:43.751019    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e34a51b1-98b4-49a7-914b-ab0949e57525-metrics-certs\") pod \"maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl\" (UID: \"e34a51b1-98b4-49a7-914b-ab0949e57525\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"
Sep 15 19:04:43.767382 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:43.767360    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqk99\" (UniqueName: \"kubernetes.io/projected/e34a51b1-98b4-49a7-914b-ab0949e57525-kube-api-access-rqk99\") pod \"maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl\" (UID: \"e34a51b1-98b4-49a7-914b-ab0949e57525\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"
Sep 15 19:04:44.253379 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:44.253339    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e34a51b1-98b4-49a7-914b-ab0949e57525-maas-api-tls\") pod \"maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl\" (UID: \"e34a51b1-98b4-49a7-914b-ab0949e57525\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"
Sep 15 19:04:44.255801 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:44.255783    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e34a51b1-98b4-49a7-914b-ab0949e57525-maas-api-tls\") pod \"maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl\" (UID: \"e34a51b1-98b4-49a7-914b-ab0949e57525\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"
Sep 15 19:04:44.555362 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:44.555260    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"
Sep 15 19:04:44.826411 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:44.826385    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"]
Sep 15 19:04:44.828711 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:04:44.828678    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34a51b1_98b4_49a7_914b_ab0949e57525.slice/crio-67ffb3db277b954265986ee5af04dcb9b8ff1bf06d14ecabe47067c1c56e86a8 WatchSource:0}: Error finding container 67ffb3db277b954265986ee5af04dcb9b8ff1bf06d14ecabe47067c1c56e86a8: Status 404 returned error can't find the container with id 67ffb3db277b954265986ee5af04dcb9b8ff1bf06d14ecabe47067c1c56e86a8
Sep 15 19:04:45.039358 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:45.039319    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl" event={"ID":"e34a51b1-98b4-49a7-914b-ab0949e57525","Type":"ContainerStarted","Data":"67ffb3db277b954265986ee5af04dcb9b8ff1bf06d14ecabe47067c1c56e86a8"}
Sep 15 19:04:46.041454 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:46.041420    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl" event={"ID":"e34a51b1-98b4-49a7-914b-ab0949e57525","Type":"ContainerStarted","Data":"0ec7d30398e2aab87c27e303a9cd82a974e65ca99079ae14c82b9f9742c278f2"}
Sep 15 19:04:46.041869 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:46.041528    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"
Sep 15 19:04:46.060240 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:46.060193    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl" podStartSLOduration=2.728081393 podStartE2EDuration="3.060174548s" totalImagesPullingTime="332.093155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:04:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:04:46.058779104 +0000 UTC m=+827.546108090" watchObservedRunningTime="2026-09-15 19:04:46.060174548 +0000 UTC m=+827.547503532"
Sep 15 19:04:52.052745 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:52.052717    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"
Sep 15 19:04:57.148156 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:57.148121    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"]
Sep 15 19:04:57.148747 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:57.148336    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq" podUID="50d30e09-6ad4-4750-b08b-23bb45ffd12f" containerName="maas-api" containerID="cri-o://bebbfeb30928715e06c80602c7d4dc9fc1e12c34f7acb785b5a674ab070a860f" gracePeriod=30
Sep 15 19:04:57.402201 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:57.402148    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"
Sep 15 19:04:57.479627 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:57.479584    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50d30e09-6ad4-4750-b08b-23bb45ffd12f-maas-api-tls\") pod \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\" (UID: \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\") "
Sep 15 19:04:57.479803 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:57.479668    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5h565\" (UniqueName: \"kubernetes.io/projected/50d30e09-6ad4-4750-b08b-23bb45ffd12f-kube-api-access-5h565\") pod \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\" (UID: \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\") "
Sep 15 19:04:57.479803 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:57.479701    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d30e09-6ad4-4750-b08b-23bb45ffd12f-openshift-service-ca\") pod \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\" (UID: \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\") "
Sep 15 19:04:57.479935 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:57.479821    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50d30e09-6ad4-4750-b08b-23bb45ffd12f-metrics-certs\") pod \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\" (UID: \"50d30e09-6ad4-4750-b08b-23bb45ffd12f\") "
Sep 15 19:04:57.480094 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:57.480052    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50d30e09-6ad4-4750-b08b-23bb45ffd12f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "50d30e09-6ad4-4750-b08b-23bb45ffd12f" (UID: "50d30e09-6ad4-4750-b08b-23bb45ffd12f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:04:57.480333 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:57.480222    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d30e09-6ad4-4750-b08b-23bb45ffd12f-openshift-service-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:04:57.483490 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:57.483456    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d30e09-6ad4-4750-b08b-23bb45ffd12f-kube-api-access-5h565" (OuterVolumeSpecName: "kube-api-access-5h565") pod "50d30e09-6ad4-4750-b08b-23bb45ffd12f" (UID: "50d30e09-6ad4-4750-b08b-23bb45ffd12f"). InnerVolumeSpecName "kube-api-access-5h565". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:04:57.483490 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:57.483449    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50d30e09-6ad4-4750-b08b-23bb45ffd12f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "50d30e09-6ad4-4750-b08b-23bb45ffd12f" (UID: "50d30e09-6ad4-4750-b08b-23bb45ffd12f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:04:57.483703 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:57.483508    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50d30e09-6ad4-4750-b08b-23bb45ffd12f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "50d30e09-6ad4-4750-b08b-23bb45ffd12f" (UID: "50d30e09-6ad4-4750-b08b-23bb45ffd12f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:04:57.581296 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:57.581265    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50d30e09-6ad4-4750-b08b-23bb45ffd12f-maas-api-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:04:57.581296 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:57.581290    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h565\" (UniqueName: \"kubernetes.io/projected/50d30e09-6ad4-4750-b08b-23bb45ffd12f-kube-api-access-5h565\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:04:57.581296 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:57.581300    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50d30e09-6ad4-4750-b08b-23bb45ffd12f-metrics-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:04:58.102143 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:58.102108    2555 generic.go:356] "Generic (PLEG): container finished" podID="50d30e09-6ad4-4750-b08b-23bb45ffd12f" containerID="bebbfeb30928715e06c80602c7d4dc9fc1e12c34f7acb785b5a674ab070a860f" exitCode=0
Sep 15 19:04:58.102317 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:58.102152    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"
Sep 15 19:04:58.102317 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:58.102190    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq" event={"ID":"50d30e09-6ad4-4750-b08b-23bb45ffd12f","Type":"ContainerDied","Data":"bebbfeb30928715e06c80602c7d4dc9fc1e12c34f7acb785b5a674ab070a860f"}
Sep 15 19:04:58.102317 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:58.102238    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq" event={"ID":"50d30e09-6ad4-4750-b08b-23bb45ffd12f","Type":"ContainerDied","Data":"2d9cddfd4edcd9969fba90dac5c6fc4dd0f337e82362089c8dd48efd274f7287"}
Sep 15 19:04:58.102317 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:58.102261    2555 scope.go:117] "RemoveContainer" containerID="bebbfeb30928715e06c80602c7d4dc9fc1e12c34f7acb785b5a674ab070a860f"
Sep 15 19:04:58.111520 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:58.111505    2555 scope.go:117] "RemoveContainer" containerID="bebbfeb30928715e06c80602c7d4dc9fc1e12c34f7acb785b5a674ab070a860f"
Sep 15 19:04:58.111828 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:04:58.111809    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bebbfeb30928715e06c80602c7d4dc9fc1e12c34f7acb785b5a674ab070a860f\": container with ID starting with bebbfeb30928715e06c80602c7d4dc9fc1e12c34f7acb785b5a674ab070a860f not found: ID does not exist" containerID="bebbfeb30928715e06c80602c7d4dc9fc1e12c34f7acb785b5a674ab070a860f"
Sep 15 19:04:58.111898 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:58.111837    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebbfeb30928715e06c80602c7d4dc9fc1e12c34f7acb785b5a674ab070a860f"} err="failed to get container status \"bebbfeb30928715e06c80602c7d4dc9fc1e12c34f7acb785b5a674ab070a860f\": rpc error: code = NotFound desc = could not find container \"bebbfeb30928715e06c80602c7d4dc9fc1e12c34f7acb785b5a674ab070a860f\": container with ID starting with bebbfeb30928715e06c80602c7d4dc9fc1e12c34f7acb785b5a674ab070a860f not found: ID does not exist"
Sep 15 19:04:58.124690 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:58.124664    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"]
Sep 15 19:04:58.126833 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:58.126811    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4df819e5-7dc97bdffb-5rpfq"]
Sep 15 19:04:59.040961 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.040928    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50d30e09-6ad4-4750-b08b-23bb45ffd12f" path="/var/lib/kubelet/pods/50d30e09-6ad4-4750-b08b-23bb45ffd12f/volumes"
Sep 15 19:04:59.724053 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.724014    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"]
Sep 15 19:04:59.724465 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.724453    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d30e09-6ad4-4750-b08b-23bb45ffd12f" containerName="maas-api"
Sep 15 19:04:59.724509 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.724467    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d30e09-6ad4-4750-b08b-23bb45ffd12f" containerName="maas-api"
Sep 15 19:04:59.724551 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.724540    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d30e09-6ad4-4750-b08b-23bb45ffd12f" containerName="maas-api"
Sep 15 19:04:59.729135 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.729117    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:04:59.731591 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.731568    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7c96ca\"/\"default-dockercfg-pcpcj\""
Sep 15 19:04:59.731741 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.731594    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-7c96ca\"/\"kube-root-ca.crt\""
Sep 15 19:04:59.732396 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.732376    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-7c96ca\"/\"openshift-service-ca.crt\""
Sep 15 19:04:59.732548 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.732525    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7c96ca\"/\"simulator-w3-7c96ca-kserve-self-signed-certs\""
Sep 15 19:04:59.736962 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.736941    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"]
Sep 15 19:04:59.803136 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.803104    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-tls-certs\") pod \"simulator-w3-7c96ca-kserve-66dc48bb47-stl4c\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") " pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:04:59.803317 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.803171    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-tmp-dir\") pod \"simulator-w3-7c96ca-kserve-66dc48bb47-stl4c\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") " pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:04:59.803317 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.803198    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-model-cache\") pod \"simulator-w3-7c96ca-kserve-66dc48bb47-stl4c\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") " pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:04:59.803445 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.803322    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-home\") pod \"simulator-w3-7c96ca-kserve-66dc48bb47-stl4c\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") " pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:04:59.803445 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.803355    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw4ng\" (UniqueName: \"kubernetes.io/projected/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-kube-api-access-nw4ng\") pod \"simulator-w3-7c96ca-kserve-66dc48bb47-stl4c\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") " pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:04:59.803445 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.803407    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-dshm\") pod \"simulator-w3-7c96ca-kserve-66dc48bb47-stl4c\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") " pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:04:59.904120 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.904086    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-home\") pod \"simulator-w3-7c96ca-kserve-66dc48bb47-stl4c\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") " pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:04:59.904120 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.904129    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw4ng\" (UniqueName: \"kubernetes.io/projected/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-kube-api-access-nw4ng\") pod \"simulator-w3-7c96ca-kserve-66dc48bb47-stl4c\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") " pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:04:59.904381 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.904154    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-dshm\") pod \"simulator-w3-7c96ca-kserve-66dc48bb47-stl4c\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") " pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:04:59.904381 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.904199    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-tls-certs\") pod \"simulator-w3-7c96ca-kserve-66dc48bb47-stl4c\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") " pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:04:59.904381 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.904236    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-tmp-dir\") pod \"simulator-w3-7c96ca-kserve-66dc48bb47-stl4c\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") " pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:04:59.904381 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.904255    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-model-cache\") pod \"simulator-w3-7c96ca-kserve-66dc48bb47-stl4c\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") " pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:04:59.904592 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.904559    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-home\") pod \"simulator-w3-7c96ca-kserve-66dc48bb47-stl4c\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") " pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:04:59.904698 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.904673    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-tmp-dir\") pod \"simulator-w3-7c96ca-kserve-66dc48bb47-stl4c\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") " pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:04:59.904740 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.904708    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-model-cache\") pod \"simulator-w3-7c96ca-kserve-66dc48bb47-stl4c\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") " pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:04:59.906543 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.906510    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-dshm\") pod \"simulator-w3-7c96ca-kserve-66dc48bb47-stl4c\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") " pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:04:59.906741 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.906723    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-tls-certs\") pod \"simulator-w3-7c96ca-kserve-66dc48bb47-stl4c\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") " pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:04:59.911444 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:04:59.911422    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw4ng\" (UniqueName: \"kubernetes.io/projected/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-kube-api-access-nw4ng\") pod \"simulator-w3-7c96ca-kserve-66dc48bb47-stl4c\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") " pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:05:00.039685 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:00.039649    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:05:00.314201 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:00.314032    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"]
Sep 15 19:05:00.318892 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:05:00.318864    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cb4e1f6_31d0_4fab_9f08_1dd91c819725.slice/crio-5ae03f48bb1dd62c78db885a19b0101cd1e167d4a7dda0841dd15c3e983a659f WatchSource:0}: Error finding container 5ae03f48bb1dd62c78db885a19b0101cd1e167d4a7dda0841dd15c3e983a659f: Status 404 returned error can't find the container with id 5ae03f48bb1dd62c78db885a19b0101cd1e167d4a7dda0841dd15c3e983a659f
Sep 15 19:05:01.120557 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:01.120517    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c" event={"ID":"2cb4e1f6-31d0-4fab-9f08-1dd91c819725","Type":"ContainerStarted","Data":"82641bde1e236c15f7e1b68e54337621d85b2be35af5dbd24eff72471146d224"}
Sep 15 19:05:01.120748 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:01.120567    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c" event={"ID":"2cb4e1f6-31d0-4fab-9f08-1dd91c819725","Type":"ContainerStarted","Data":"5ae03f48bb1dd62c78db885a19b0101cd1e167d4a7dda0841dd15c3e983a659f"}
Sep 15 19:05:01.144298 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:01.144252    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c" podStartSLOduration=2.144236289 podStartE2EDuration="2.144236289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:05:01.142279771 +0000 UTC m=+842.629608759" watchObservedRunningTime="2026-09-15 19:05:01.144236289 +0000 UTC m=+842.631565274"
Sep 15 19:05:05.642514 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:05.642480    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"]
Sep 15 19:05:05.646775 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:05.646752    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"
Sep 15 19:05:05.649049 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:05.649027    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-813c61ca-serving-cert\""
Sep 15 19:05:05.659658 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:05.659627    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"]
Sep 15 19:05:05.764714 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:05.764676    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033f5b61-7e47-4082-8883-97218a9d6372-maas-api-tls\") pod \"maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2\" (UID: \"033f5b61-7e47-4082-8883-97218a9d6372\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"
Sep 15 19:05:05.764919 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:05.764744    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033f5b61-7e47-4082-8883-97218a9d6372-metrics-certs\") pod \"maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2\" (UID: \"033f5b61-7e47-4082-8883-97218a9d6372\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"
Sep 15 19:05:05.764919 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:05.764824    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7zbr\" (UniqueName: \"kubernetes.io/projected/033f5b61-7e47-4082-8883-97218a9d6372-kube-api-access-z7zbr\") pod \"maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2\" (UID: \"033f5b61-7e47-4082-8883-97218a9d6372\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"
Sep 15 19:05:05.764919 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:05.764887    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033f5b61-7e47-4082-8883-97218a9d6372-openshift-service-ca\") pod \"maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2\" (UID: \"033f5b61-7e47-4082-8883-97218a9d6372\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"
Sep 15 19:05:05.865874 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:05.865835    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033f5b61-7e47-4082-8883-97218a9d6372-maas-api-tls\") pod \"maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2\" (UID: \"033f5b61-7e47-4082-8883-97218a9d6372\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"
Sep 15 19:05:05.866067 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:05.865892    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033f5b61-7e47-4082-8883-97218a9d6372-metrics-certs\") pod \"maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2\" (UID: \"033f5b61-7e47-4082-8883-97218a9d6372\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"
Sep 15 19:05:05.866067 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:05.865916    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7zbr\" (UniqueName: \"kubernetes.io/projected/033f5b61-7e47-4082-8883-97218a9d6372-kube-api-access-z7zbr\") pod \"maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2\" (UID: \"033f5b61-7e47-4082-8883-97218a9d6372\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"
Sep 15 19:05:05.866067 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:05.865949    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033f5b61-7e47-4082-8883-97218a9d6372-openshift-service-ca\") pod \"maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2\" (UID: \"033f5b61-7e47-4082-8883-97218a9d6372\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"
Sep 15 19:05:05.866575 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:05.866554    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033f5b61-7e47-4082-8883-97218a9d6372-openshift-service-ca\") pod \"maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2\" (UID: \"033f5b61-7e47-4082-8883-97218a9d6372\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"
Sep 15 19:05:05.868251 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:05.868224    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033f5b61-7e47-4082-8883-97218a9d6372-metrics-certs\") pod \"maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2\" (UID: \"033f5b61-7e47-4082-8883-97218a9d6372\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"
Sep 15 19:05:05.868384 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:05.868365    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033f5b61-7e47-4082-8883-97218a9d6372-maas-api-tls\") pod \"maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2\" (UID: \"033f5b61-7e47-4082-8883-97218a9d6372\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"
Sep 15 19:05:05.876431 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:05.876410    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7zbr\" (UniqueName: \"kubernetes.io/projected/033f5b61-7e47-4082-8883-97218a9d6372-kube-api-access-z7zbr\") pod \"maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2\" (UID: \"033f5b61-7e47-4082-8883-97218a9d6372\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"
Sep 15 19:05:05.962261 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:05.962170    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"
Sep 15 19:05:06.234647 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:06.234603    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"]
Sep 15 19:05:06.236688 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:05:06.236660    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033f5b61_7e47_4082_8883_97218a9d6372.slice/crio-d7ae4b42af28a7972939bfe832b69fde1274d18431662ad491e6e5819df5397f WatchSource:0}: Error finding container d7ae4b42af28a7972939bfe832b69fde1274d18431662ad491e6e5819df5397f: Status 404 returned error can't find the container with id d7ae4b42af28a7972939bfe832b69fde1274d18431662ad491e6e5819df5397f
Sep 15 19:05:07.152174 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:07.152132    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2" event={"ID":"033f5b61-7e47-4082-8883-97218a9d6372","Type":"ContainerStarted","Data":"50a9d36de9c6dccd6ac6a3aeb54e4d972cc3b07240c0bd09c3d845ec13837472"}
Sep 15 19:05:07.152585 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:07.152202    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2" event={"ID":"033f5b61-7e47-4082-8883-97218a9d6372","Type":"ContainerStarted","Data":"d7ae4b42af28a7972939bfe832b69fde1274d18431662ad491e6e5819df5397f"}
Sep 15 19:05:07.152585 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:07.152234    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"
Sep 15 19:05:07.167790 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:07.167743    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2" podStartSLOduration=1.739275788 podStartE2EDuration="2.167725957s" totalImagesPullingTime="428.450169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:05:07.16693229 +0000 UTC m=+848.654261276" watchObservedRunningTime="2026-09-15 19:05:07.167725957 +0000 UTC m=+848.655054942"
Sep 15 19:05:10.040391 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:10.040345    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:05:10.040391 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:10.040400    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:05:10.053523 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:10.053492    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:05:10.180385 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:10.180357    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:05:13.163455 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:13.163426    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"
Sep 15 19:05:16.621162 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.621111    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"]
Sep 15 19:05:16.625185 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.625161    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:16.627352 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.627332    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-74ef1f\"/\"openshift-service-ca.crt\""
Sep 15 19:05:16.627473 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.627378    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-74ef1f\"/\"simulator-w0-74ef1f-kserve-self-signed-certs\""
Sep 15 19:05:16.628400 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.628381    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-74ef1f\"/\"default-dockercfg-grz98\""
Sep 15 19:05:16.628515 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.628413    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-74ef1f\"/\"kube-root-ca.crt\""
Sep 15 19:05:16.633947 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.633927    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"]
Sep 15 19:05:16.777757 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.777719    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-model-cache\") pod \"simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") " pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:16.777943 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.777766    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-tmp-dir\") pod \"simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") " pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:16.777943 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.777847    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-dshm\") pod \"simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") " pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:16.777943 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.777902    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a312e92a-cbd1-4cea-add8-eb83f2d48a71-tls-certs\") pod \"simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") " pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:16.777943 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.777932    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-home\") pod \"simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") " pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:16.778115 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.777982    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6lk7\" (UniqueName: \"kubernetes.io/projected/a312e92a-cbd1-4cea-add8-eb83f2d48a71-kube-api-access-c6lk7\") pod \"simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") " pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:16.879359 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.879251    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-tmp-dir\") pod \"simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") " pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:16.879359 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.879328    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-dshm\") pod \"simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") " pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:16.879642 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.879374    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a312e92a-cbd1-4cea-add8-eb83f2d48a71-tls-certs\") pod \"simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") " pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:16.879642 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.879412    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-home\") pod \"simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") " pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:16.879642 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.879464    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6lk7\" (UniqueName: \"kubernetes.io/projected/a312e92a-cbd1-4cea-add8-eb83f2d48a71-kube-api-access-c6lk7\") pod \"simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") " pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:16.879642 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.879522    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-model-cache\") pod \"simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") " pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:16.879875 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.879687    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-tmp-dir\") pod \"simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") " pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:16.879875 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.879800    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-home\") pod \"simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") " pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:16.879972 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.879895    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-model-cache\") pod \"simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") " pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:16.881507 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.881486    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-dshm\") pod \"simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") " pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:16.881863 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.881846    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a312e92a-cbd1-4cea-add8-eb83f2d48a71-tls-certs\") pod \"simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") " pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:16.886691 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.886673    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6lk7\" (UniqueName: \"kubernetes.io/projected/a312e92a-cbd1-4cea-add8-eb83f2d48a71-kube-api-access-c6lk7\") pod \"simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") " pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:16.936256 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:16.936213    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:17.207099 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:17.207072    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"]
Sep 15 19:05:17.208922 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:05:17.208896    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda312e92a_cbd1_4cea_add8_eb83f2d48a71.slice/crio-1bb42a892eb7dec2b9357447e12aed0a8fbe6b1c379cf9cf7122051ad6072307 WatchSource:0}: Error finding container 1bb42a892eb7dec2b9357447e12aed0a8fbe6b1c379cf9cf7122051ad6072307: Status 404 returned error can't find the container with id 1bb42a892eb7dec2b9357447e12aed0a8fbe6b1c379cf9cf7122051ad6072307
Sep 15 19:05:18.213472 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:18.213432    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g" event={"ID":"a312e92a-cbd1-4cea-add8-eb83f2d48a71","Type":"ContainerStarted","Data":"8cb88b872b15e9b46e7820d202b17dcf1ca7eb02f064349c78ad934b1dadb803"}
Sep 15 19:05:18.213472 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:18.213478    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g" event={"ID":"a312e92a-cbd1-4cea-add8-eb83f2d48a71","Type":"ContainerStarted","Data":"1bb42a892eb7dec2b9357447e12aed0a8fbe6b1c379cf9cf7122051ad6072307"}
Sep 15 19:05:18.231379 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:18.231311    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g" podStartSLOduration=2.231293998 podStartE2EDuration="2.231293998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:05:18.228760798 +0000 UTC m=+859.716089784" watchObservedRunningTime="2026-09-15 19:05:18.231293998 +0000 UTC m=+859.718622993"
Sep 15 19:05:26.936902 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:26.936868    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:26.937325 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:26.936918    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:26.949843 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:26.949806    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:27.285371 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:27.285339    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:05:39.442129 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.442092    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"]
Sep 15 19:05:39.445838 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.445817    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:39.449981 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.449952    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d5559a\"/\"openshift-service-ca.crt\""
Sep 15 19:05:39.450123 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.450036    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d5559a\"/\"test-model-d5559a-kserve-self-signed-certs\""
Sep 15 19:05:39.450123 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.449962    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d5559a\"/\"kube-root-ca.crt\""
Sep 15 19:05:39.450347 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.450322    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d5559a\"/\"default-dockercfg-npnph\""
Sep 15 19:05:39.453017 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.452994    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"]
Sep 15 19:05:39.498049 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.498013    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7504140-a834-43d2-a00a-38d6fce32411-tls-certs\") pod \"test-model-d5559a-kserve-85bd446c7f-2gpcn\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") " pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:39.498049 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.498052    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6kqn\" (UniqueName: \"kubernetes.io/projected/a7504140-a834-43d2-a00a-38d6fce32411-kube-api-access-h6kqn\") pod \"test-model-d5559a-kserve-85bd446c7f-2gpcn\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") " pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:39.498334 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.498160    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-tmp-dir\") pod \"test-model-d5559a-kserve-85bd446c7f-2gpcn\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") " pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:39.498334 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.498217    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-model-cache\") pod \"test-model-d5559a-kserve-85bd446c7f-2gpcn\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") " pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:39.498334 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.498284    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-dshm\") pod \"test-model-d5559a-kserve-85bd446c7f-2gpcn\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") " pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:39.498334 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.498326    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-home\") pod \"test-model-d5559a-kserve-85bd446c7f-2gpcn\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") " pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:39.599822 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.599779    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7504140-a834-43d2-a00a-38d6fce32411-tls-certs\") pod \"test-model-d5559a-kserve-85bd446c7f-2gpcn\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") " pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:39.599822 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.599821    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6kqn\" (UniqueName: \"kubernetes.io/projected/a7504140-a834-43d2-a00a-38d6fce32411-kube-api-access-h6kqn\") pod \"test-model-d5559a-kserve-85bd446c7f-2gpcn\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") " pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:39.600072 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.599860    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-tmp-dir\") pod \"test-model-d5559a-kserve-85bd446c7f-2gpcn\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") " pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:39.600072 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.599886    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-model-cache\") pod \"test-model-d5559a-kserve-85bd446c7f-2gpcn\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") " pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:39.600072 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.600026    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-dshm\") pod \"test-model-d5559a-kserve-85bd446c7f-2gpcn\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") " pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:39.600225 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.600074    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-home\") pod \"test-model-d5559a-kserve-85bd446c7f-2gpcn\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") " pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:39.600295 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.600274    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-model-cache\") pod \"test-model-d5559a-kserve-85bd446c7f-2gpcn\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") " pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:39.600343 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.600322    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-tmp-dir\") pod \"test-model-d5559a-kserve-85bd446c7f-2gpcn\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") " pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:39.600429 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.600410    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-home\") pod \"test-model-d5559a-kserve-85bd446c7f-2gpcn\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") " pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:39.602250 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.602232    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-dshm\") pod \"test-model-d5559a-kserve-85bd446c7f-2gpcn\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") " pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:39.602438 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.602421    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7504140-a834-43d2-a00a-38d6fce32411-tls-certs\") pod \"test-model-d5559a-kserve-85bd446c7f-2gpcn\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") " pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:39.607480 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.607457    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6kqn\" (UniqueName: \"kubernetes.io/projected/a7504140-a834-43d2-a00a-38d6fce32411-kube-api-access-h6kqn\") pod \"test-model-d5559a-kserve-85bd446c7f-2gpcn\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") " pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:39.757764 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:39.757728    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:40.034426 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:40.034398    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"]
Sep 15 19:05:40.036563 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:05:40.036537    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7504140_a834_43d2_a00a_38d6fce32411.slice/crio-f197506a8b5521fed181b6973b4c1c617c6f4e0a917ce5c11dc15c984c907e05 WatchSource:0}: Error finding container f197506a8b5521fed181b6973b4c1c617c6f4e0a917ce5c11dc15c984c907e05: Status 404 returned error can't find the container with id f197506a8b5521fed181b6973b4c1c617c6f4e0a917ce5c11dc15c984c907e05
Sep 15 19:05:40.342729 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:40.342633    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn" event={"ID":"a7504140-a834-43d2-a00a-38d6fce32411","Type":"ContainerStarted","Data":"74177f6e40b4e7c5752a30ef0354e913eca373534a493d113157fa93804ca576"}
Sep 15 19:05:40.342729 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:40.342685    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn" event={"ID":"a7504140-a834-43d2-a00a-38d6fce32411","Type":"ContainerStarted","Data":"f197506a8b5521fed181b6973b4c1c617c6f4e0a917ce5c11dc15c984c907e05"}
Sep 15 19:05:43.172202 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:43.172147    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn" podStartSLOduration=4.172129267 podStartE2EDuration="4.172129267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:05:40.362303367 +0000 UTC m=+881.849632353" watchObservedRunningTime="2026-09-15 19:05:43.172129267 +0000 UTC m=+884.659458253"
Sep 15 19:05:43.173473 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:43.173449    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"]
Sep 15 19:05:43.173714 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:43.173685    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2" podUID="033f5b61-7e47-4082-8883-97218a9d6372" containerName="maas-api" containerID="cri-o://50a9d36de9c6dccd6ac6a3aeb54e4d972cc3b07240c0bd09c3d845ec13837472" gracePeriod=30
Sep 15 19:05:43.363141 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:43.363098    2555 generic.go:356] "Generic (PLEG): container finished" podID="033f5b61-7e47-4082-8883-97218a9d6372" containerID="50a9d36de9c6dccd6ac6a3aeb54e4d972cc3b07240c0bd09c3d845ec13837472" exitCode=0
Sep 15 19:05:43.363330 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:43.363220    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2" event={"ID":"033f5b61-7e47-4082-8883-97218a9d6372","Type":"ContainerDied","Data":"50a9d36de9c6dccd6ac6a3aeb54e4d972cc3b07240c0bd09c3d845ec13837472"}
Sep 15 19:05:43.420858 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:43.420830    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"
Sep 15 19:05:43.538623 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:43.538581    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7zbr\" (UniqueName: \"kubernetes.io/projected/033f5b61-7e47-4082-8883-97218a9d6372-kube-api-access-z7zbr\") pod \"033f5b61-7e47-4082-8883-97218a9d6372\" (UID: \"033f5b61-7e47-4082-8883-97218a9d6372\") "
Sep 15 19:05:43.538808 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:43.538649    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033f5b61-7e47-4082-8883-97218a9d6372-maas-api-tls\") pod \"033f5b61-7e47-4082-8883-97218a9d6372\" (UID: \"033f5b61-7e47-4082-8883-97218a9d6372\") "
Sep 15 19:05:43.538808 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:43.538728    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033f5b61-7e47-4082-8883-97218a9d6372-metrics-certs\") pod \"033f5b61-7e47-4082-8883-97218a9d6372\" (UID: \"033f5b61-7e47-4082-8883-97218a9d6372\") "
Sep 15 19:05:43.538808 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:43.538770    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033f5b61-7e47-4082-8883-97218a9d6372-openshift-service-ca\") pod \"033f5b61-7e47-4082-8883-97218a9d6372\" (UID: \"033f5b61-7e47-4082-8883-97218a9d6372\") "
Sep 15 19:05:43.539200 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:43.539164    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033f5b61-7e47-4082-8883-97218a9d6372-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "033f5b61-7e47-4082-8883-97218a9d6372" (UID: "033f5b61-7e47-4082-8883-97218a9d6372"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:05:43.541095 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:43.541040    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/033f5b61-7e47-4082-8883-97218a9d6372-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "033f5b61-7e47-4082-8883-97218a9d6372" (UID: "033f5b61-7e47-4082-8883-97218a9d6372"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:05:43.541329 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:43.541306    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/033f5b61-7e47-4082-8883-97218a9d6372-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "033f5b61-7e47-4082-8883-97218a9d6372" (UID: "033f5b61-7e47-4082-8883-97218a9d6372"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:05:43.541654 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:43.541633    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033f5b61-7e47-4082-8883-97218a9d6372-kube-api-access-z7zbr" (OuterVolumeSpecName: "kube-api-access-z7zbr") pod "033f5b61-7e47-4082-8883-97218a9d6372" (UID: "033f5b61-7e47-4082-8883-97218a9d6372"). InnerVolumeSpecName "kube-api-access-z7zbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:05:43.639896 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:43.639859    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033f5b61-7e47-4082-8883-97218a9d6372-openshift-service-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:05:43.639896 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:43.639889    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7zbr\" (UniqueName: \"kubernetes.io/projected/033f5b61-7e47-4082-8883-97218a9d6372-kube-api-access-z7zbr\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:05:43.639896 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:43.639899    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033f5b61-7e47-4082-8883-97218a9d6372-maas-api-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:05:43.640129 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:43.639910    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033f5b61-7e47-4082-8883-97218a9d6372-metrics-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:05:44.370047 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:44.370010    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"
Sep 15 19:05:44.370564 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:44.370041    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2" event={"ID":"033f5b61-7e47-4082-8883-97218a9d6372","Type":"ContainerDied","Data":"d7ae4b42af28a7972939bfe832b69fde1274d18431662ad491e6e5819df5397f"}
Sep 15 19:05:44.370564 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:44.370102    2555 scope.go:117] "RemoveContainer" containerID="50a9d36de9c6dccd6ac6a3aeb54e4d972cc3b07240c0bd09c3d845ec13837472"
Sep 15 19:05:44.390760 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:44.390735    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"]
Sep 15 19:05:44.394277 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:44.394254    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-813c61ca-65d7dd4887-tf7j2"]
Sep 15 19:05:45.040683 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:45.040654    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="033f5b61-7e47-4082-8883-97218a9d6372" path="/var/lib/kubelet/pods/033f5b61-7e47-4082-8883-97218a9d6372/volumes"
Sep 15 19:05:49.758025 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:49.757985    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:49.758502 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:49.758050    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:49.770844 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:49.770817    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:49.841443 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:49.841411    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"]
Sep 15 19:05:49.842067 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:49.842048    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="033f5b61-7e47-4082-8883-97218a9d6372" containerName="maas-api"
Sep 15 19:05:49.842067 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:49.842067    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="033f5b61-7e47-4082-8883-97218a9d6372" containerName="maas-api"
Sep 15 19:05:49.842238 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:49.842207    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="033f5b61-7e47-4082-8883-97218a9d6372" containerName="maas-api"
Sep 15 19:05:49.847048 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:49.847025    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"
Sep 15 19:05:49.849176 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:49.849154    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e886bcb7-serving-cert\""
Sep 15 19:05:49.855929 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:49.855895    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"]
Sep 15 19:05:50.003620 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:50.003578    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a502a7df-77e5-47b9-8b32-2e4f3101eed4-openshift-service-ca\") pod \"maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5\" (UID: \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"
Sep 15 19:05:50.003802 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:50.003689    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a502a7df-77e5-47b9-8b32-2e4f3101eed4-metrics-certs\") pod \"maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5\" (UID: \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"
Sep 15 19:05:50.003802 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:50.003784    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlhgd\" (UniqueName: \"kubernetes.io/projected/a502a7df-77e5-47b9-8b32-2e4f3101eed4-kube-api-access-vlhgd\") pod \"maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5\" (UID: \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"
Sep 15 19:05:50.003881 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:50.003837    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a502a7df-77e5-47b9-8b32-2e4f3101eed4-maas-api-tls\") pod \"maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5\" (UID: \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"
Sep 15 19:05:50.104455 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:50.104372    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a502a7df-77e5-47b9-8b32-2e4f3101eed4-maas-api-tls\") pod \"maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5\" (UID: \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"
Sep 15 19:05:50.104455 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:50.104444    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a502a7df-77e5-47b9-8b32-2e4f3101eed4-openshift-service-ca\") pod \"maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5\" (UID: \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"
Sep 15 19:05:50.104719 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:50.104475    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a502a7df-77e5-47b9-8b32-2e4f3101eed4-metrics-certs\") pod \"maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5\" (UID: \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"
Sep 15 19:05:50.104719 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:50.104529    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlhgd\" (UniqueName: \"kubernetes.io/projected/a502a7df-77e5-47b9-8b32-2e4f3101eed4-kube-api-access-vlhgd\") pod \"maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5\" (UID: \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"
Sep 15 19:05:50.105126 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:50.105105    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a502a7df-77e5-47b9-8b32-2e4f3101eed4-openshift-service-ca\") pod \"maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5\" (UID: \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"
Sep 15 19:05:50.106854 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:50.106834    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a502a7df-77e5-47b9-8b32-2e4f3101eed4-metrics-certs\") pod \"maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5\" (UID: \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"
Sep 15 19:05:50.106911 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:50.106890    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a502a7df-77e5-47b9-8b32-2e4f3101eed4-maas-api-tls\") pod \"maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5\" (UID: \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"
Sep 15 19:05:50.113726 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:50.113698    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlhgd\" (UniqueName: \"kubernetes.io/projected/a502a7df-77e5-47b9-8b32-2e4f3101eed4-kube-api-access-vlhgd\") pod \"maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5\" (UID: \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"
Sep 15 19:05:50.183535 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:50.183502    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"
Sep 15 19:05:50.411399 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:50.411311    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:05:50.457659 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:50.457626    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"]
Sep 15 19:05:50.460002 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:05:50.459972    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda502a7df_77e5_47b9_8b32_2e4f3101eed4.slice/crio-9c3eccb5881cefd144b52df0c7760932878556fe28fd04beac6c45f71ba52ea6 WatchSource:0}: Error finding container 9c3eccb5881cefd144b52df0c7760932878556fe28fd04beac6c45f71ba52ea6: Status 404 returned error can't find the container with id 9c3eccb5881cefd144b52df0c7760932878556fe28fd04beac6c45f71ba52ea6
Sep 15 19:05:51.406904 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:51.406868    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5" event={"ID":"a502a7df-77e5-47b9-8b32-2e4f3101eed4","Type":"ContainerStarted","Data":"bbc66431da5d4a29f3023700c475f1728a69d01ed2d30b7c062f2a9e5dd70886"}
Sep 15 19:05:51.407300 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:51.406922    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5" event={"ID":"a502a7df-77e5-47b9-8b32-2e4f3101eed4","Type":"ContainerStarted","Data":"9c3eccb5881cefd144b52df0c7760932878556fe28fd04beac6c45f71ba52ea6"}
Sep 15 19:05:51.407300 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:51.407074    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"
Sep 15 19:05:51.425290 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:51.425240    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5" podStartSLOduration=2.045760775 podStartE2EDuration="2.425224601s" totalImagesPullingTime="379.463826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:05:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:05:51.423167614 +0000 UTC m=+892.910496598" watchObservedRunningTime="2026-09-15 19:05:51.425224601 +0000 UTC m=+892.912553635"
Sep 15 19:05:57.417317 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:05:57.417286    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"
Sep 15 19:06:04.652946 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.652908    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"]
Sep 15 19:06:04.658420 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.658399    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:04.660763 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.660738    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7c96ca\"/\"premium-w3-7c96ca-kserve-self-signed-certs\""
Sep 15 19:06:04.664900 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.664872    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"]
Sep 15 19:06:04.748838 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.748798    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-tmp-dir\") pod \"premium-w3-7c96ca-kserve-844d8d98-kpqpq\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") " pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:04.749041 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.748868    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-model-cache\") pod \"premium-w3-7c96ca-kserve-844d8d98-kpqpq\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") " pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:04.749041 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.748887    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-home\") pod \"premium-w3-7c96ca-kserve-844d8d98-kpqpq\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") " pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:04.749041 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.748904    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw86w\" (UniqueName: \"kubernetes.io/projected/0169d288-9a3a-4738-a6e6-922210087864-kube-api-access-mw86w\") pod \"premium-w3-7c96ca-kserve-844d8d98-kpqpq\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") " pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:04.749041 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.748937    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-dshm\") pod \"premium-w3-7c96ca-kserve-844d8d98-kpqpq\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") " pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:04.749247 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.749083    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0169d288-9a3a-4738-a6e6-922210087864-tls-certs\") pod \"premium-w3-7c96ca-kserve-844d8d98-kpqpq\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") " pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:04.850176 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.850133    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0169d288-9a3a-4738-a6e6-922210087864-tls-certs\") pod \"premium-w3-7c96ca-kserve-844d8d98-kpqpq\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") " pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:04.850176 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.850170    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-tmp-dir\") pod \"premium-w3-7c96ca-kserve-844d8d98-kpqpq\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") " pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:04.850434 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.850224    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-model-cache\") pod \"premium-w3-7c96ca-kserve-844d8d98-kpqpq\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") " pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:04.850434 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.850255    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-home\") pod \"premium-w3-7c96ca-kserve-844d8d98-kpqpq\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") " pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:04.850434 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.850274    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw86w\" (UniqueName: \"kubernetes.io/projected/0169d288-9a3a-4738-a6e6-922210087864-kube-api-access-mw86w\") pod \"premium-w3-7c96ca-kserve-844d8d98-kpqpq\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") " pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:04.850434 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.850321    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-dshm\") pod \"premium-w3-7c96ca-kserve-844d8d98-kpqpq\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") " pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:04.850918 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.850894    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-tmp-dir\") pod \"premium-w3-7c96ca-kserve-844d8d98-kpqpq\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") " pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:04.851078 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.850910    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-home\") pod \"premium-w3-7c96ca-kserve-844d8d98-kpqpq\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") " pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:04.851078 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.850956    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-model-cache\") pod \"premium-w3-7c96ca-kserve-844d8d98-kpqpq\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") " pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:04.853229 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.853205    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-dshm\") pod \"premium-w3-7c96ca-kserve-844d8d98-kpqpq\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") " pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:04.853377 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.853358    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0169d288-9a3a-4738-a6e6-922210087864-tls-certs\") pod \"premium-w3-7c96ca-kserve-844d8d98-kpqpq\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") " pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:04.858174 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.858147    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw86w\" (UniqueName: \"kubernetes.io/projected/0169d288-9a3a-4738-a6e6-922210087864-kube-api-access-mw86w\") pod \"premium-w3-7c96ca-kserve-844d8d98-kpqpq\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") " pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:04.970380 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:04.970296    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:05.241297 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:05.241271    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"]
Sep 15 19:06:05.480746 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:05.480704    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq" event={"ID":"0169d288-9a3a-4738-a6e6-922210087864","Type":"ContainerStarted","Data":"ae14218c1f1a316754a0b193db3b3eb97e708b07ead8fca9cf2f3cb25401244d"}
Sep 15 19:06:05.480914 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:05.480765    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq" event={"ID":"0169d288-9a3a-4738-a6e6-922210087864","Type":"ContainerStarted","Data":"65bcf8e661f660d573e9c5e06d19431b76158c29adccf6b5303dcdb40adfa82c"}
Sep 15 19:06:05.497259 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:05.497204    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq" podStartSLOduration=1.497187322 podStartE2EDuration="1.497187322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:06:05.49549301 +0000 UTC m=+906.982822000" watchObservedRunningTime="2026-09-15 19:06:05.497187322 +0000 UTC m=+906.984516306"
Sep 15 19:06:14.970706 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:14.970671    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:14.973235 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:14.970762    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:14.983460 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:14.983439    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:15.542428 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:15.542394    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:06:28.361139 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:28.361098    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"]
Sep 15 19:06:28.361707 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:28.361494    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5" podUID="a502a7df-77e5-47b9-8b32-2e4f3101eed4" containerName="maas-api" containerID="cri-o://bbc66431da5d4a29f3023700c475f1728a69d01ed2d30b7c062f2a9e5dd70886" gracePeriod=30
Sep 15 19:06:28.601952 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:28.601923    2555 generic.go:356] "Generic (PLEG): container finished" podID="a502a7df-77e5-47b9-8b32-2e4f3101eed4" containerID="bbc66431da5d4a29f3023700c475f1728a69d01ed2d30b7c062f2a9e5dd70886" exitCode=0
Sep 15 19:06:28.602091 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:28.601984    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5" event={"ID":"a502a7df-77e5-47b9-8b32-2e4f3101eed4","Type":"ContainerDied","Data":"bbc66431da5d4a29f3023700c475f1728a69d01ed2d30b7c062f2a9e5dd70886"}
Sep 15 19:06:28.602091 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:28.602033    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5" event={"ID":"a502a7df-77e5-47b9-8b32-2e4f3101eed4","Type":"ContainerDied","Data":"9c3eccb5881cefd144b52df0c7760932878556fe28fd04beac6c45f71ba52ea6"}
Sep 15 19:06:28.602091 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:28.602056    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c3eccb5881cefd144b52df0c7760932878556fe28fd04beac6c45f71ba52ea6"
Sep 15 19:06:28.610076 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:28.610054    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"
Sep 15 19:06:28.699582 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:28.699497    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlhgd\" (UniqueName: \"kubernetes.io/projected/a502a7df-77e5-47b9-8b32-2e4f3101eed4-kube-api-access-vlhgd\") pod \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\" (UID: \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\") "
Sep 15 19:06:28.699756 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:28.699580    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a502a7df-77e5-47b9-8b32-2e4f3101eed4-openshift-service-ca\") pod \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\" (UID: \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\") "
Sep 15 19:06:28.699756 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:28.699687    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a502a7df-77e5-47b9-8b32-2e4f3101eed4-metrics-certs\") pod \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\" (UID: \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\") "
Sep 15 19:06:28.699756 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:28.699728    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a502a7df-77e5-47b9-8b32-2e4f3101eed4-maas-api-tls\") pod \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\" (UID: \"a502a7df-77e5-47b9-8b32-2e4f3101eed4\") "
Sep 15 19:06:28.700017 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:28.699985    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a502a7df-77e5-47b9-8b32-2e4f3101eed4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a502a7df-77e5-47b9-8b32-2e4f3101eed4" (UID: "a502a7df-77e5-47b9-8b32-2e4f3101eed4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:06:28.700175 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:28.700156    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a502a7df-77e5-47b9-8b32-2e4f3101eed4-openshift-service-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:06:28.701859 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:28.701832    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a502a7df-77e5-47b9-8b32-2e4f3101eed4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a502a7df-77e5-47b9-8b32-2e4f3101eed4" (UID: "a502a7df-77e5-47b9-8b32-2e4f3101eed4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:06:28.701995 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:28.701916    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a502a7df-77e5-47b9-8b32-2e4f3101eed4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a502a7df-77e5-47b9-8b32-2e4f3101eed4" (UID: "a502a7df-77e5-47b9-8b32-2e4f3101eed4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:06:28.701995 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:28.701923    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a502a7df-77e5-47b9-8b32-2e4f3101eed4-kube-api-access-vlhgd" (OuterVolumeSpecName: "kube-api-access-vlhgd") pod "a502a7df-77e5-47b9-8b32-2e4f3101eed4" (UID: "a502a7df-77e5-47b9-8b32-2e4f3101eed4"). InnerVolumeSpecName "kube-api-access-vlhgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:06:28.801405 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:28.801369    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a502a7df-77e5-47b9-8b32-2e4f3101eed4-metrics-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:06:28.801405 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:28.801400    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a502a7df-77e5-47b9-8b32-2e4f3101eed4-maas-api-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:06:28.801405 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:28.801411    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlhgd\" (UniqueName: \"kubernetes.io/projected/a502a7df-77e5-47b9-8b32-2e4f3101eed4-kube-api-access-vlhgd\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:06:29.606749 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:29.606708    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"
Sep 15 19:06:29.621810 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:29.621783    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"]
Sep 15 19:06:29.624884 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:29.624857    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e886bcb7-777bbdb57f-bwrm5"]
Sep 15 19:06:31.040913 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:31.040883    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a502a7df-77e5-47b9-8b32-2e4f3101eed4" path="/var/lib/kubelet/pods/a502a7df-77e5-47b9-8b32-2e4f3101eed4/volumes"
Sep 15 19:06:36.239423 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.239391    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"]
Sep 15 19:06:36.239947 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.239928    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a502a7df-77e5-47b9-8b32-2e4f3101eed4" containerName="maas-api"
Sep 15 19:06:36.239947 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.239948    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a502a7df-77e5-47b9-8b32-2e4f3101eed4" containerName="maas-api"
Sep 15 19:06:36.240074 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.240051    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a502a7df-77e5-47b9-8b32-2e4f3101eed4" containerName="maas-api"
Sep 15 19:06:36.244009 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.243987    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"
Sep 15 19:06:36.246351 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.246332    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-b6741af4-serving-cert\""
Sep 15 19:06:36.255004 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.254978    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"]
Sep 15 19:06:36.274086 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.274059    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3edd2fa-9851-4338-a356-4baf092a8b11-openshift-service-ca\") pod \"maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj\" (UID: \"d3edd2fa-9851-4338-a356-4baf092a8b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"
Sep 15 19:06:36.274223 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.274100    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3edd2fa-9851-4338-a356-4baf092a8b11-metrics-certs\") pod \"maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj\" (UID: \"d3edd2fa-9851-4338-a356-4baf092a8b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"
Sep 15 19:06:36.274223 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.274184    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p52n2\" (UniqueName: \"kubernetes.io/projected/d3edd2fa-9851-4338-a356-4baf092a8b11-kube-api-access-p52n2\") pod \"maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj\" (UID: \"d3edd2fa-9851-4338-a356-4baf092a8b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"
Sep 15 19:06:36.274223 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.274219    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3edd2fa-9851-4338-a356-4baf092a8b11-maas-api-tls\") pod \"maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj\" (UID: \"d3edd2fa-9851-4338-a356-4baf092a8b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"
Sep 15 19:06:36.374853 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.374819    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3edd2fa-9851-4338-a356-4baf092a8b11-metrics-certs\") pod \"maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj\" (UID: \"d3edd2fa-9851-4338-a356-4baf092a8b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"
Sep 15 19:06:36.375028 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.374963    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p52n2\" (UniqueName: \"kubernetes.io/projected/d3edd2fa-9851-4338-a356-4baf092a8b11-kube-api-access-p52n2\") pod \"maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj\" (UID: \"d3edd2fa-9851-4338-a356-4baf092a8b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"
Sep 15 19:06:36.375028 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.374997    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3edd2fa-9851-4338-a356-4baf092a8b11-maas-api-tls\") pod \"maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj\" (UID: \"d3edd2fa-9851-4338-a356-4baf092a8b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"
Sep 15 19:06:36.375111 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.375044    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3edd2fa-9851-4338-a356-4baf092a8b11-openshift-service-ca\") pod \"maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj\" (UID: \"d3edd2fa-9851-4338-a356-4baf092a8b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"
Sep 15 19:06:36.375151 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:06:36.375126    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-serving-cert: secret "maas-api-e2e-derive-b6741af4-serving-cert" not found
Sep 15 19:06:36.375228 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:06:36.375214    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3edd2fa-9851-4338-a356-4baf092a8b11-maas-api-tls podName:d3edd2fa-9851-4338-a356-4baf092a8b11 nodeName:}" failed. No retries permitted until 2026-09-15 19:06:36.875193263 +0000 UTC m=+938.362522232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3edd2fa-9851-4338-a356-4baf092a8b11-maas-api-tls") pod "maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj" (UID: "d3edd2fa-9851-4338-a356-4baf092a8b11") : secret "maas-api-e2e-derive-b6741af4-serving-cert" not found
Sep 15 19:06:36.375762 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.375744    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3edd2fa-9851-4338-a356-4baf092a8b11-openshift-service-ca\") pod \"maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj\" (UID: \"d3edd2fa-9851-4338-a356-4baf092a8b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"
Sep 15 19:06:36.377255 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.377237    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3edd2fa-9851-4338-a356-4baf092a8b11-metrics-certs\") pod \"maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj\" (UID: \"d3edd2fa-9851-4338-a356-4baf092a8b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"
Sep 15 19:06:36.385320 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.385294    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p52n2\" (UniqueName: \"kubernetes.io/projected/d3edd2fa-9851-4338-a356-4baf092a8b11-kube-api-access-p52n2\") pod \"maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj\" (UID: \"d3edd2fa-9851-4338-a356-4baf092a8b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"
Sep 15 19:06:36.880920 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.880887    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3edd2fa-9851-4338-a356-4baf092a8b11-maas-api-tls\") pod \"maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj\" (UID: \"d3edd2fa-9851-4338-a356-4baf092a8b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"
Sep 15 19:06:36.883238 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:36.883214    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3edd2fa-9851-4338-a356-4baf092a8b11-maas-api-tls\") pod \"maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj\" (UID: \"d3edd2fa-9851-4338-a356-4baf092a8b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"
Sep 15 19:06:37.157879 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:37.157782    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"
Sep 15 19:06:37.428047 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:37.428015    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"]
Sep 15 19:06:37.430225 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:06:37.430196    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3edd2fa_9851_4338_a356_4baf092a8b11.slice/crio-c2c12259371023ae87592df8ced8084801b0231ed81c7f9bb1c946e6bb5e20c5 WatchSource:0}: Error finding container c2c12259371023ae87592df8ced8084801b0231ed81c7f9bb1c946e6bb5e20c5: Status 404 returned error can't find the container with id c2c12259371023ae87592df8ced8084801b0231ed81c7f9bb1c946e6bb5e20c5
Sep 15 19:06:37.649252 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:37.649217    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj" event={"ID":"d3edd2fa-9851-4338-a356-4baf092a8b11","Type":"ContainerStarted","Data":"c2c12259371023ae87592df8ced8084801b0231ed81c7f9bb1c946e6bb5e20c5"}
Sep 15 19:06:38.657620 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:38.657586    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj" event={"ID":"d3edd2fa-9851-4338-a356-4baf092a8b11","Type":"ContainerStarted","Data":"72f071f5356712c2e1a26e0f15dbf34c0e8d0776a3c6696acb7cf0950eb95e9c"}
Sep 15 19:06:38.658000 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:38.657783    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"
Sep 15 19:06:38.675294 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:38.675224    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj" podStartSLOduration=2.160629695 podStartE2EDuration="2.675206427s" totalImagesPullingTime="514.576732ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:06:38.673683277 +0000 UTC m=+940.161012263" watchObservedRunningTime="2026-09-15 19:06:38.675206427 +0000 UTC m=+940.162535411"
Sep 15 19:06:44.668480 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:06:44.668448    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"
Sep 15 19:07:04.717284 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:04.717246    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"]
Sep 15 19:07:04.717912 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:04.717515    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj" podUID="d3edd2fa-9851-4338-a356-4baf092a8b11" containerName="maas-api" containerID="cri-o://72f071f5356712c2e1a26e0f15dbf34c0e8d0776a3c6696acb7cf0950eb95e9c" gracePeriod=30
Sep 15 19:07:04.975725 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:04.975669    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"
Sep 15 19:07:04.979122 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:04.979097    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3edd2fa-9851-4338-a356-4baf092a8b11-metrics-certs\") pod \"d3edd2fa-9851-4338-a356-4baf092a8b11\" (UID: \"d3edd2fa-9851-4338-a356-4baf092a8b11\") "
Sep 15 19:07:04.979235 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:04.979219    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p52n2\" (UniqueName: \"kubernetes.io/projected/d3edd2fa-9851-4338-a356-4baf092a8b11-kube-api-access-p52n2\") pod \"d3edd2fa-9851-4338-a356-4baf092a8b11\" (UID: \"d3edd2fa-9851-4338-a356-4baf092a8b11\") "
Sep 15 19:07:04.979295 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:04.979287    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3edd2fa-9851-4338-a356-4baf092a8b11-maas-api-tls\") pod \"d3edd2fa-9851-4338-a356-4baf092a8b11\" (UID: \"d3edd2fa-9851-4338-a356-4baf092a8b11\") "
Sep 15 19:07:04.979365 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:04.979350    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3edd2fa-9851-4338-a356-4baf092a8b11-openshift-service-ca\") pod \"d3edd2fa-9851-4338-a356-4baf092a8b11\" (UID: \"d3edd2fa-9851-4338-a356-4baf092a8b11\") "
Sep 15 19:07:04.979708 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:04.979682    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3edd2fa-9851-4338-a356-4baf092a8b11-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3edd2fa-9851-4338-a356-4baf092a8b11" (UID: "d3edd2fa-9851-4338-a356-4baf092a8b11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:07:04.981213 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:04.981189    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3edd2fa-9851-4338-a356-4baf092a8b11-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d3edd2fa-9851-4338-a356-4baf092a8b11" (UID: "d3edd2fa-9851-4338-a356-4baf092a8b11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:07:04.981722 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:04.981703    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3edd2fa-9851-4338-a356-4baf092a8b11-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3edd2fa-9851-4338-a356-4baf092a8b11" (UID: "d3edd2fa-9851-4338-a356-4baf092a8b11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:07:04.981789 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:04.981750    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3edd2fa-9851-4338-a356-4baf092a8b11-kube-api-access-p52n2" (OuterVolumeSpecName: "kube-api-access-p52n2") pod "d3edd2fa-9851-4338-a356-4baf092a8b11" (UID: "d3edd2fa-9851-4338-a356-4baf092a8b11"). InnerVolumeSpecName "kube-api-access-p52n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:07:05.081500 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:05.081471    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3edd2fa-9851-4338-a356-4baf092a8b11-metrics-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:07:05.081697 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:05.081659    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p52n2\" (UniqueName: \"kubernetes.io/projected/d3edd2fa-9851-4338-a356-4baf092a8b11-kube-api-access-p52n2\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:07:05.081697 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:05.081678    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3edd2fa-9851-4338-a356-4baf092a8b11-maas-api-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:07:05.081815 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:05.081703    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3edd2fa-9851-4338-a356-4baf092a8b11-openshift-service-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:07:05.804666 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:05.804628    2555 generic.go:356] "Generic (PLEG): container finished" podID="d3edd2fa-9851-4338-a356-4baf092a8b11" containerID="72f071f5356712c2e1a26e0f15dbf34c0e8d0776a3c6696acb7cf0950eb95e9c" exitCode=0
Sep 15 19:07:05.805072 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:05.804688    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"
Sep 15 19:07:05.805072 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:05.804702    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj" event={"ID":"d3edd2fa-9851-4338-a356-4baf092a8b11","Type":"ContainerDied","Data":"72f071f5356712c2e1a26e0f15dbf34c0e8d0776a3c6696acb7cf0950eb95e9c"}
Sep 15 19:07:05.805072 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:05.804785    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj" event={"ID":"d3edd2fa-9851-4338-a356-4baf092a8b11","Type":"ContainerDied","Data":"c2c12259371023ae87592df8ced8084801b0231ed81c7f9bb1c946e6bb5e20c5"}
Sep 15 19:07:05.805072 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:05.804807    2555 scope.go:117] "RemoveContainer" containerID="72f071f5356712c2e1a26e0f15dbf34c0e8d0776a3c6696acb7cf0950eb95e9c"
Sep 15 19:07:05.814093 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:05.814081    2555 scope.go:117] "RemoveContainer" containerID="72f071f5356712c2e1a26e0f15dbf34c0e8d0776a3c6696acb7cf0950eb95e9c"
Sep 15 19:07:05.814374 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:07:05.814356    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f071f5356712c2e1a26e0f15dbf34c0e8d0776a3c6696acb7cf0950eb95e9c\": container with ID starting with 72f071f5356712c2e1a26e0f15dbf34c0e8d0776a3c6696acb7cf0950eb95e9c not found: ID does not exist" containerID="72f071f5356712c2e1a26e0f15dbf34c0e8d0776a3c6696acb7cf0950eb95e9c"
Sep 15 19:07:05.814421 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:05.814382    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f071f5356712c2e1a26e0f15dbf34c0e8d0776a3c6696acb7cf0950eb95e9c"} err="failed to get container status \"72f071f5356712c2e1a26e0f15dbf34c0e8d0776a3c6696acb7cf0950eb95e9c\": rpc error: code = NotFound desc = could not find container \"72f071f5356712c2e1a26e0f15dbf34c0e8d0776a3c6696acb7cf0950eb95e9c\": container with ID starting with 72f071f5356712c2e1a26e0f15dbf34c0e8d0776a3c6696acb7cf0950eb95e9c not found: ID does not exist"
Sep 15 19:07:05.821258 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:05.821235    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"]
Sep 15 19:07:05.828255 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:05.828235    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b6741af4-74b87899b8-xlqtj"]
Sep 15 19:07:07.040498 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:07.040467    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3edd2fa-9851-4338-a356-4baf092a8b11" path="/var/lib/kubelet/pods/d3edd2fa-9851-4338-a356-4baf092a8b11/volumes"
Sep 15 19:07:14.247149 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.247108    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"]
Sep 15 19:07:14.247812 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.247794    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3edd2fa-9851-4338-a356-4baf092a8b11" containerName="maas-api"
Sep 15 19:07:14.247867 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.247816    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3edd2fa-9851-4338-a356-4baf092a8b11" containerName="maas-api"
Sep 15 19:07:14.247958 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.247946    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3edd2fa-9851-4338-a356-4baf092a8b11" containerName="maas-api"
Sep 15 19:07:14.252692 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.252661    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.255787 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.255763    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7c96ca\"/\"distinct-w3-7c96ca-kserve-self-signed-certs\""
Sep 15 19:07:14.262686 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.262593    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"]
Sep 15 19:07:14.269099 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.269076    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f81db17b-7991-4f3f-a51d-45158ff78d14-tls-certs\") pod \"distinct-w3-7c96ca-kserve-6c45675b77-kr79d\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.269209 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.269118    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-home\") pod \"distinct-w3-7c96ca-kserve-6c45675b77-kr79d\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.269209 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.269158    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-dshm\") pod \"distinct-w3-7c96ca-kserve-6c45675b77-kr79d\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.269209 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.269202    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmz4g\" (UniqueName: \"kubernetes.io/projected/f81db17b-7991-4f3f-a51d-45158ff78d14-kube-api-access-bmz4g\") pod \"distinct-w3-7c96ca-kserve-6c45675b77-kr79d\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.269323 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.269257    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-tmp-dir\") pod \"distinct-w3-7c96ca-kserve-6c45675b77-kr79d\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.269323 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.269275    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-model-cache\") pod \"distinct-w3-7c96ca-kserve-6c45675b77-kr79d\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.370450 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.370421    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmz4g\" (UniqueName: \"kubernetes.io/projected/f81db17b-7991-4f3f-a51d-45158ff78d14-kube-api-access-bmz4g\") pod \"distinct-w3-7c96ca-kserve-6c45675b77-kr79d\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.370738 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.370470    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-tmp-dir\") pod \"distinct-w3-7c96ca-kserve-6c45675b77-kr79d\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.370738 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.370492    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-model-cache\") pod \"distinct-w3-7c96ca-kserve-6c45675b77-kr79d\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.370738 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.370561    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f81db17b-7991-4f3f-a51d-45158ff78d14-tls-certs\") pod \"distinct-w3-7c96ca-kserve-6c45675b77-kr79d\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.370738 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.370594    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-home\") pod \"distinct-w3-7c96ca-kserve-6c45675b77-kr79d\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.370738 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.370650    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-dshm\") pod \"distinct-w3-7c96ca-kserve-6c45675b77-kr79d\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.370975 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.370895    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-tmp-dir\") pod \"distinct-w3-7c96ca-kserve-6c45675b77-kr79d\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.370975 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.370962    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-model-cache\") pod \"distinct-w3-7c96ca-kserve-6c45675b77-kr79d\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.371061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.371042    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-home\") pod \"distinct-w3-7c96ca-kserve-6c45675b77-kr79d\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.372801 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.372782    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-dshm\") pod \"distinct-w3-7c96ca-kserve-6c45675b77-kr79d\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.373063 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.373045    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f81db17b-7991-4f3f-a51d-45158ff78d14-tls-certs\") pod \"distinct-w3-7c96ca-kserve-6c45675b77-kr79d\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.378163 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.378139    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmz4g\" (UniqueName: \"kubernetes.io/projected/f81db17b-7991-4f3f-a51d-45158ff78d14-kube-api-access-bmz4g\") pod \"distinct-w3-7c96ca-kserve-6c45675b77-kr79d\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") " pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.565313 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.565237    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:14.835306 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.835276    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"]
Sep 15 19:07:14.836311 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:07:14.836276    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf81db17b_7991_4f3f_a51d_45158ff78d14.slice/crio-0e8b512d69e09b49989a39208edfd5a04ebbb14e564e4d78008af09c3c2c69ea WatchSource:0}: Error finding container 0e8b512d69e09b49989a39208edfd5a04ebbb14e564e4d78008af09c3c2c69ea: Status 404 returned error can't find the container with id 0e8b512d69e09b49989a39208edfd5a04ebbb14e564e4d78008af09c3c2c69ea
Sep 15 19:07:14.855154 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:14.855127    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d" event={"ID":"f81db17b-7991-4f3f-a51d-45158ff78d14","Type":"ContainerStarted","Data":"0e8b512d69e09b49989a39208edfd5a04ebbb14e564e4d78008af09c3c2c69ea"}
Sep 15 19:07:15.862993 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:15.862958    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d" event={"ID":"f81db17b-7991-4f3f-a51d-45158ff78d14","Type":"ContainerStarted","Data":"41e49e80b5629b7367ff83ce446f10917567bc87c9a2e19f8d35231a2c5394e8"}
Sep 15 19:07:15.880752 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:15.880704    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d" podStartSLOduration=1.880689909 podStartE2EDuration="1.880689909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:15.878361358 +0000 UTC m=+977.365690344" watchObservedRunningTime="2026-09-15 19:07:15.880689909 +0000 UTC m=+977.368018994"
Sep 15 19:07:24.565728 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:24.565691    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:24.566087 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:24.565939    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:24.578624 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:24.578590    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:07:24.928380 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:07:24.928297    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:08:47.350022 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:47.349993    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"]
Sep 15 19:08:47.357977 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:47.357949    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"
Sep 15 19:08:47.368553 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:47.364951    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-84ee0c4c-serving-cert\""
Sep 15 19:08:47.369452 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:47.369384    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"]
Sep 15 19:08:47.523831 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:47.523786    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27xpc\" (UniqueName: \"kubernetes.io/projected/7dde93dc-0113-45c7-a961-e6d3956c5465-kube-api-access-27xpc\") pod \"maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs\" (UID: \"7dde93dc-0113-45c7-a961-e6d3956c5465\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"
Sep 15 19:08:47.524041 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:47.523840    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dde93dc-0113-45c7-a961-e6d3956c5465-openshift-service-ca\") pod \"maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs\" (UID: \"7dde93dc-0113-45c7-a961-e6d3956c5465\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"
Sep 15 19:08:47.524041 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:47.523930    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dde93dc-0113-45c7-a961-e6d3956c5465-metrics-certs\") pod \"maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs\" (UID: \"7dde93dc-0113-45c7-a961-e6d3956c5465\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"
Sep 15 19:08:47.524041 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:47.523992    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dde93dc-0113-45c7-a961-e6d3956c5465-maas-api-tls\") pod \"maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs\" (UID: \"7dde93dc-0113-45c7-a961-e6d3956c5465\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"
Sep 15 19:08:47.624780 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:47.624681    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dde93dc-0113-45c7-a961-e6d3956c5465-metrics-certs\") pod \"maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs\" (UID: \"7dde93dc-0113-45c7-a961-e6d3956c5465\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"
Sep 15 19:08:47.624780 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:47.624761    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dde93dc-0113-45c7-a961-e6d3956c5465-maas-api-tls\") pod \"maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs\" (UID: \"7dde93dc-0113-45c7-a961-e6d3956c5465\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"
Sep 15 19:08:47.625016 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:47.624867    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27xpc\" (UniqueName: \"kubernetes.io/projected/7dde93dc-0113-45c7-a961-e6d3956c5465-kube-api-access-27xpc\") pod \"maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs\" (UID: \"7dde93dc-0113-45c7-a961-e6d3956c5465\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"
Sep 15 19:08:47.625016 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:08:47.624894    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-serving-cert: secret "maas-api-e2e-mt-84ee0c4c-serving-cert" not found
Sep 15 19:08:47.625016 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:08:47.624972    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dde93dc-0113-45c7-a961-e6d3956c5465-maas-api-tls podName:7dde93dc-0113-45c7-a961-e6d3956c5465 nodeName:}" failed. No retries permitted until 2026-09-15 19:08:48.124955102 +0000 UTC m=+1069.612284065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7dde93dc-0113-45c7-a961-e6d3956c5465-maas-api-tls") pod "maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs" (UID: "7dde93dc-0113-45c7-a961-e6d3956c5465") : secret "maas-api-e2e-mt-84ee0c4c-serving-cert" not found
Sep 15 19:08:47.625016 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:47.624903    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dde93dc-0113-45c7-a961-e6d3956c5465-openshift-service-ca\") pod \"maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs\" (UID: \"7dde93dc-0113-45c7-a961-e6d3956c5465\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"
Sep 15 19:08:47.625543 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:47.625520    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dde93dc-0113-45c7-a961-e6d3956c5465-openshift-service-ca\") pod \"maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs\" (UID: \"7dde93dc-0113-45c7-a961-e6d3956c5465\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"
Sep 15 19:08:47.627129 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:47.627108    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dde93dc-0113-45c7-a961-e6d3956c5465-metrics-certs\") pod \"maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs\" (UID: \"7dde93dc-0113-45c7-a961-e6d3956c5465\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"
Sep 15 19:08:47.635463 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:47.635441    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27xpc\" (UniqueName: \"kubernetes.io/projected/7dde93dc-0113-45c7-a961-e6d3956c5465-kube-api-access-27xpc\") pod \"maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs\" (UID: \"7dde93dc-0113-45c7-a961-e6d3956c5465\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"
Sep 15 19:08:48.129907 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:48.129848    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dde93dc-0113-45c7-a961-e6d3956c5465-maas-api-tls\") pod \"maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs\" (UID: \"7dde93dc-0113-45c7-a961-e6d3956c5465\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"
Sep 15 19:08:48.132311 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:48.132285    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dde93dc-0113-45c7-a961-e6d3956c5465-maas-api-tls\") pod \"maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs\" (UID: \"7dde93dc-0113-45c7-a961-e6d3956c5465\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"
Sep 15 19:08:48.282250 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:48.282207    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"
Sep 15 19:08:48.554660 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:48.554632    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"]
Sep 15 19:08:48.556759 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:08:48.556732    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dde93dc_0113_45c7_a961_e6d3956c5465.slice/crio-b7a423374ba17bb28e3be40eb5cc15bd613d2d0c84b908ce3a54e508eb67e50e WatchSource:0}: Error finding container b7a423374ba17bb28e3be40eb5cc15bd613d2d0c84b908ce3a54e508eb67e50e: Status 404 returned error can't find the container with id b7a423374ba17bb28e3be40eb5cc15bd613d2d0c84b908ce3a54e508eb67e50e
Sep 15 19:08:48.558874 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:48.558853    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:08:49.371994 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:49.371894    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs" event={"ID":"7dde93dc-0113-45c7-a961-e6d3956c5465","Type":"ContainerStarted","Data":"22a5fe0b12e8d237495ce527851a25a6b3f20fcaceacfd37e1f840eda613a76f"}
Sep 15 19:08:49.371994 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:49.371965    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs" event={"ID":"7dde93dc-0113-45c7-a961-e6d3956c5465","Type":"ContainerStarted","Data":"b7a423374ba17bb28e3be40eb5cc15bd613d2d0c84b908ce3a54e508eb67e50e"}
Sep 15 19:08:49.372202 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:49.371998    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"
Sep 15 19:08:49.386921 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:49.386868    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs" podStartSLOduration=1.930622855 podStartE2EDuration="2.386851025s" totalImagesPullingTime="456.22817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:08:49.386544357 +0000 UTC m=+1070.873873370" watchObservedRunningTime="2026-09-15 19:08:49.386851025 +0000 UTC m=+1070.874180010"
Sep 15 19:08:55.383370 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:08:55.383296    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"
Sep 15 19:09:08.926741 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:08.926699    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"]
Sep 15 19:09:08.930510 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:08.930488    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:08.932834 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:08.932817    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-74ef1f\"/\"distinct-w0-74ef1f-kserve-self-signed-certs\""
Sep 15 19:09:08.939256 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:08.939234    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"]
Sep 15 19:09:09.039369 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.039341    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-model-cache\") pod \"distinct-w0-74ef1f-kserve-5487894456-9s92v\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:09.039538 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.039377    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-dshm\") pod \"distinct-w0-74ef1f-kserve-5487894456-9s92v\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:09.039538 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.039395    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-tmp-dir\") pod \"distinct-w0-74ef1f-kserve-5487894456-9s92v\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:09.039538 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.039420    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26830a72-a1ff-42e4-9693-d40ca366462e-tls-certs\") pod \"distinct-w0-74ef1f-kserve-5487894456-9s92v\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:09.039538 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.039514    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqsc6\" (UniqueName: \"kubernetes.io/projected/26830a72-a1ff-42e4-9693-d40ca366462e-kube-api-access-kqsc6\") pod \"distinct-w0-74ef1f-kserve-5487894456-9s92v\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:09.039719 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.039541    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-home\") pod \"distinct-w0-74ef1f-kserve-5487894456-9s92v\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:09.140469 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.140436    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-model-cache\") pod \"distinct-w0-74ef1f-kserve-5487894456-9s92v\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:09.140662 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.140486    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-dshm\") pod \"distinct-w0-74ef1f-kserve-5487894456-9s92v\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:09.140662 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.140510    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-tmp-dir\") pod \"distinct-w0-74ef1f-kserve-5487894456-9s92v\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:09.140662 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.140538    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26830a72-a1ff-42e4-9693-d40ca366462e-tls-certs\") pod \"distinct-w0-74ef1f-kserve-5487894456-9s92v\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:09.140662 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.140596    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqsc6\" (UniqueName: \"kubernetes.io/projected/26830a72-a1ff-42e4-9693-d40ca366462e-kube-api-access-kqsc6\") pod \"distinct-w0-74ef1f-kserve-5487894456-9s92v\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:09.140662 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.140646    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-home\") pod \"distinct-w0-74ef1f-kserve-5487894456-9s92v\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:09.141011 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.140908    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-model-cache\") pod \"distinct-w0-74ef1f-kserve-5487894456-9s92v\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:09.141011 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.140981    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-tmp-dir\") pod \"distinct-w0-74ef1f-kserve-5487894456-9s92v\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:09.141011 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.141008    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-home\") pod \"distinct-w0-74ef1f-kserve-5487894456-9s92v\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:09.142858 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.142833    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-dshm\") pod \"distinct-w0-74ef1f-kserve-5487894456-9s92v\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:09.143050 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.143030    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26830a72-a1ff-42e4-9693-d40ca366462e-tls-certs\") pod \"distinct-w0-74ef1f-kserve-5487894456-9s92v\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:09.148382 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.148357    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqsc6\" (UniqueName: \"kubernetes.io/projected/26830a72-a1ff-42e4-9693-d40ca366462e-kube-api-access-kqsc6\") pod \"distinct-w0-74ef1f-kserve-5487894456-9s92v\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:09.240883 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.240798    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:09.519085 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:09.519056    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"]
Sep 15 19:09:09.521170 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:09:09.521122    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26830a72_a1ff_42e4_9693_d40ca366462e.slice/crio-c97b4d0070608ced5db145d897715710eac6912d99b9f0132469d97e9381f4f5 WatchSource:0}: Error finding container c97b4d0070608ced5db145d897715710eac6912d99b9f0132469d97e9381f4f5: Status 404 returned error can't find the container with id c97b4d0070608ced5db145d897715710eac6912d99b9f0132469d97e9381f4f5
Sep 15 19:09:10.484253 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:10.484220    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v" event={"ID":"26830a72-a1ff-42e4-9693-d40ca366462e","Type":"ContainerStarted","Data":"5c50f900675c627a0f5f2246de68e723894c7c77b01f13dd3cfa06182aaef5ab"}
Sep 15 19:09:10.484253 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:10.484260    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v" event={"ID":"26830a72-a1ff-42e4-9693-d40ca366462e","Type":"ContainerStarted","Data":"c97b4d0070608ced5db145d897715710eac6912d99b9f0132469d97e9381f4f5"}
Sep 15 19:09:10.502501 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:10.502454    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v" podStartSLOduration=2.502435395 podStartE2EDuration="2.502435395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:09:10.499865176 +0000 UTC m=+1091.987194172" watchObservedRunningTime="2026-09-15 19:09:10.502435395 +0000 UTC m=+1091.989764380"
Sep 15 19:09:13.015862 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:13.015829    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"]
Sep 15 19:09:19.241757 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:19.241698    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:19.241757 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:19.241768    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:19.257633 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:19.257577    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:19.547600 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:19.547567    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:09:26.083593 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.083556    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"]
Sep 15 19:09:26.084025 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.083873    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr" podUID="cc25a84c-4256-44f4-8f51-3cd1dc33cf80" containerName="maas-api" containerID="cri-o://e2cb69da4e26a2e5e1255f642387fd6cb00e7de007bfe588355b9a335e745ae6" gracePeriod=30
Sep 15 19:09:26.345116 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.345052    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"
Sep 15 19:09:26.522961 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.522922    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-maas-api-tls\") pod \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\" (UID: \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\") "
Sep 15 19:09:26.522961 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.522966    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrtl8\" (UniqueName: \"kubernetes.io/projected/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-kube-api-access-jrtl8\") pod \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\" (UID: \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\") "
Sep 15 19:09:26.523221 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.523026    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-metrics-certs\") pod \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\" (UID: \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\") "
Sep 15 19:09:26.523221 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.523059    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-openshift-service-ca\") pod \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\" (UID: \"cc25a84c-4256-44f4-8f51-3cd1dc33cf80\") "
Sep 15 19:09:26.523471 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.523444    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc25a84c-4256-44f4-8f51-3cd1dc33cf80" (UID: "cc25a84c-4256-44f4-8f51-3cd1dc33cf80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:09:26.525048 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.525017    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc25a84c-4256-44f4-8f51-3cd1dc33cf80" (UID: "cc25a84c-4256-44f4-8f51-3cd1dc33cf80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:09:26.525144 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.525067    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-kube-api-access-jrtl8" (OuterVolumeSpecName: "kube-api-access-jrtl8") pod "cc25a84c-4256-44f4-8f51-3cd1dc33cf80" (UID: "cc25a84c-4256-44f4-8f51-3cd1dc33cf80"). InnerVolumeSpecName "kube-api-access-jrtl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:26.525144 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.525102    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cc25a84c-4256-44f4-8f51-3cd1dc33cf80" (UID: "cc25a84c-4256-44f4-8f51-3cd1dc33cf80"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:09:26.576425 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.576393    2555 generic.go:356] "Generic (PLEG): container finished" podID="cc25a84c-4256-44f4-8f51-3cd1dc33cf80" containerID="e2cb69da4e26a2e5e1255f642387fd6cb00e7de007bfe588355b9a335e745ae6" exitCode=0
Sep 15 19:09:26.576592 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.576438    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"
Sep 15 19:09:26.576592 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.576479    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr" event={"ID":"cc25a84c-4256-44f4-8f51-3cd1dc33cf80","Type":"ContainerDied","Data":"e2cb69da4e26a2e5e1255f642387fd6cb00e7de007bfe588355b9a335e745ae6"}
Sep 15 19:09:26.576592 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.576529    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr" event={"ID":"cc25a84c-4256-44f4-8f51-3cd1dc33cf80","Type":"ContainerDied","Data":"71a246c24f9e00a84d587d601d1947cad8cd4dea84119b0487d4f6c1a05b821c"}
Sep 15 19:09:26.576592 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.576551    2555 scope.go:117] "RemoveContainer" containerID="e2cb69da4e26a2e5e1255f642387fd6cb00e7de007bfe588355b9a335e745ae6"
Sep 15 19:09:26.585838 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.585818    2555 scope.go:117] "RemoveContainer" containerID="e2cb69da4e26a2e5e1255f642387fd6cb00e7de007bfe588355b9a335e745ae6"
Sep 15 19:09:26.586140 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:09:26.586119    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2cb69da4e26a2e5e1255f642387fd6cb00e7de007bfe588355b9a335e745ae6\": container with ID starting with e2cb69da4e26a2e5e1255f642387fd6cb00e7de007bfe588355b9a335e745ae6 not found: ID does not exist" containerID="e2cb69da4e26a2e5e1255f642387fd6cb00e7de007bfe588355b9a335e745ae6"
Sep 15 19:09:26.586190 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.586150    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2cb69da4e26a2e5e1255f642387fd6cb00e7de007bfe588355b9a335e745ae6"} err="failed to get container status \"e2cb69da4e26a2e5e1255f642387fd6cb00e7de007bfe588355b9a335e745ae6\": rpc error: code = NotFound desc = could not find container \"e2cb69da4e26a2e5e1255f642387fd6cb00e7de007bfe588355b9a335e745ae6\": container with ID starting with e2cb69da4e26a2e5e1255f642387fd6cb00e7de007bfe588355b9a335e745ae6 not found: ID does not exist"
Sep 15 19:09:26.597916 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.597835    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"]
Sep 15 19:09:26.599849 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.599828    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d5559a-78d897cb8-rn8hr"]
Sep 15 19:09:26.624625 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.624590    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-metrics-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:09:26.624625 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.624631    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-openshift-service-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:09:26.624810 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.624641    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-maas-api-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:09:26.624810 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:26.624650    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrtl8\" (UniqueName: \"kubernetes.io/projected/cc25a84c-4256-44f4-8f51-3cd1dc33cf80-kube-api-access-jrtl8\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:09:27.040916 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:27.040882    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc25a84c-4256-44f4-8f51-3cd1dc33cf80" path="/var/lib/kubelet/pods/cc25a84c-4256-44f4-8f51-3cd1dc33cf80/volumes"
Sep 15 19:09:28.035993 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.035948    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn" podUID="a7504140-a834-43d2-a00a-38d6fce32411" containerName="main" containerID="cri-o://74177f6e40b4e7c5752a30ef0354e913eca373534a493d113157fa93804ca576" gracePeriod=45
Sep 15 19:09:28.291457 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.291397    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:09:28.342784 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.342755    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-model-cache\") pod \"a7504140-a834-43d2-a00a-38d6fce32411\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") "
Sep 15 19:09:28.342965 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.342798    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7504140-a834-43d2-a00a-38d6fce32411-tls-certs\") pod \"a7504140-a834-43d2-a00a-38d6fce32411\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") "
Sep 15 19:09:28.342965 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.342866    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-home\") pod \"a7504140-a834-43d2-a00a-38d6fce32411\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") "
Sep 15 19:09:28.342965 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.342886    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-dshm\") pod \"a7504140-a834-43d2-a00a-38d6fce32411\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") "
Sep 15 19:09:28.342965 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.342927    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-tmp-dir\") pod \"a7504140-a834-43d2-a00a-38d6fce32411\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") "
Sep 15 19:09:28.342965 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.342946    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6kqn\" (UniqueName: \"kubernetes.io/projected/a7504140-a834-43d2-a00a-38d6fce32411-kube-api-access-h6kqn\") pod \"a7504140-a834-43d2-a00a-38d6fce32411\" (UID: \"a7504140-a834-43d2-a00a-38d6fce32411\") "
Sep 15 19:09:28.343235 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.343035    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7504140-a834-43d2-a00a-38d6fce32411" (UID: "a7504140-a834-43d2-a00a-38d6fce32411"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:28.343235 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.343062    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-home" (OuterVolumeSpecName: "home") pod "a7504140-a834-43d2-a00a-38d6fce32411" (UID: "a7504140-a834-43d2-a00a-38d6fce32411"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:28.343235 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.343212    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7504140-a834-43d2-a00a-38d6fce32411" (UID: "a7504140-a834-43d2-a00a-38d6fce32411"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:28.343347 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.343267    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-home\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:09:28.343347 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.343281    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-model-cache\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:09:28.345134 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.345112    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-dshm" (OuterVolumeSpecName: "dshm") pod "a7504140-a834-43d2-a00a-38d6fce32411" (UID: "a7504140-a834-43d2-a00a-38d6fce32411"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:28.345237 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.345199    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7504140-a834-43d2-a00a-38d6fce32411-kube-api-access-h6kqn" (OuterVolumeSpecName: "kube-api-access-h6kqn") pod "a7504140-a834-43d2-a00a-38d6fce32411" (UID: "a7504140-a834-43d2-a00a-38d6fce32411"). InnerVolumeSpecName "kube-api-access-h6kqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:28.345282 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.345244    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7504140-a834-43d2-a00a-38d6fce32411-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7504140-a834-43d2-a00a-38d6fce32411" (UID: "a7504140-a834-43d2-a00a-38d6fce32411"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:09:28.444217 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.444180    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-dshm\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:09:28.444217 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.444209    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7504140-a834-43d2-a00a-38d6fce32411-tmp-dir\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:09:28.444217 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.444219    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6kqn\" (UniqueName: \"kubernetes.io/projected/a7504140-a834-43d2-a00a-38d6fce32411-kube-api-access-h6kqn\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:09:28.444451 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.444229    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7504140-a834-43d2-a00a-38d6fce32411-tls-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:09:28.573660 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.573551    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"]
Sep 15 19:09:28.573869 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.573837    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs" podUID="7dde93dc-0113-45c7-a961-e6d3956c5465" containerName="maas-api" containerID="cri-o://22a5fe0b12e8d237495ce527851a25a6b3f20fcaceacfd37e1f840eda613a76f" gracePeriod=30
Sep 15 19:09:28.591064 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.591028    2555 generic.go:356] "Generic (PLEG): container finished" podID="a7504140-a834-43d2-a00a-38d6fce32411" containerID="74177f6e40b4e7c5752a30ef0354e913eca373534a493d113157fa93804ca576" exitCode=0
Sep 15 19:09:28.591300 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.591276    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn" event={"ID":"a7504140-a834-43d2-a00a-38d6fce32411","Type":"ContainerDied","Data":"74177f6e40b4e7c5752a30ef0354e913eca373534a493d113157fa93804ca576"}
Sep 15 19:09:28.591375 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.591342    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn" event={"ID":"a7504140-a834-43d2-a00a-38d6fce32411","Type":"ContainerDied","Data":"f197506a8b5521fed181b6973b4c1c617c6f4e0a917ce5c11dc15c984c907e05"}
Sep 15 19:09:28.591375 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.591370    2555 scope.go:117] "RemoveContainer" containerID="74177f6e40b4e7c5752a30ef0354e913eca373534a493d113157fa93804ca576"
Sep 15 19:09:28.591542 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.591526    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"
Sep 15 19:09:28.660153 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.659957    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"]
Sep 15 19:09:28.662562 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.662536    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d5559a/test-model-d5559a-kserve-85bd446c7f-2gpcn"]
Sep 15 19:09:28.687061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.687038    2555 scope.go:117] "RemoveContainer" containerID="74177f6e40b4e7c5752a30ef0354e913eca373534a493d113157fa93804ca576"
Sep 15 19:09:28.687444 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:09:28.687417    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74177f6e40b4e7c5752a30ef0354e913eca373534a493d113157fa93804ca576\": container with ID starting with 74177f6e40b4e7c5752a30ef0354e913eca373534a493d113157fa93804ca576 not found: ID does not exist" containerID="74177f6e40b4e7c5752a30ef0354e913eca373534a493d113157fa93804ca576"
Sep 15 19:09:28.687542 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.687451    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74177f6e40b4e7c5752a30ef0354e913eca373534a493d113157fa93804ca576"} err="failed to get container status \"74177f6e40b4e7c5752a30ef0354e913eca373534a493d113157fa93804ca576\": rpc error: code = NotFound desc = could not find container \"74177f6e40b4e7c5752a30ef0354e913eca373534a493d113157fa93804ca576\": container with ID starting with 74177f6e40b4e7c5752a30ef0354e913eca373534a493d113157fa93804ca576 not found: ID does not exist"
Sep 15 19:09:28.808206 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.808183    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"
Sep 15 19:09:28.847908 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.847829    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dde93dc-0113-45c7-a961-e6d3956c5465-maas-api-tls\") pod \"7dde93dc-0113-45c7-a961-e6d3956c5465\" (UID: \"7dde93dc-0113-45c7-a961-e6d3956c5465\") "
Sep 15 19:09:28.847908 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.847890    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dde93dc-0113-45c7-a961-e6d3956c5465-metrics-certs\") pod \"7dde93dc-0113-45c7-a961-e6d3956c5465\" (UID: \"7dde93dc-0113-45c7-a961-e6d3956c5465\") "
Sep 15 19:09:28.848132 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.847917    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27xpc\" (UniqueName: \"kubernetes.io/projected/7dde93dc-0113-45c7-a961-e6d3956c5465-kube-api-access-27xpc\") pod \"7dde93dc-0113-45c7-a961-e6d3956c5465\" (UID: \"7dde93dc-0113-45c7-a961-e6d3956c5465\") "
Sep 15 19:09:28.848132 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.847940    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dde93dc-0113-45c7-a961-e6d3956c5465-openshift-service-ca\") pod \"7dde93dc-0113-45c7-a961-e6d3956c5465\" (UID: \"7dde93dc-0113-45c7-a961-e6d3956c5465\") "
Sep 15 19:09:28.848427 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.848398    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dde93dc-0113-45c7-a961-e6d3956c5465-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7dde93dc-0113-45c7-a961-e6d3956c5465" (UID: "7dde93dc-0113-45c7-a961-e6d3956c5465"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:09:28.850222 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.850188    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dde93dc-0113-45c7-a961-e6d3956c5465-kube-api-access-27xpc" (OuterVolumeSpecName: "kube-api-access-27xpc") pod "7dde93dc-0113-45c7-a961-e6d3956c5465" (UID: "7dde93dc-0113-45c7-a961-e6d3956c5465"). InnerVolumeSpecName "kube-api-access-27xpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:28.850429 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.850397    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dde93dc-0113-45c7-a961-e6d3956c5465-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7dde93dc-0113-45c7-a961-e6d3956c5465" (UID: "7dde93dc-0113-45c7-a961-e6d3956c5465"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:09:28.850510 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.850421    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dde93dc-0113-45c7-a961-e6d3956c5465-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7dde93dc-0113-45c7-a961-e6d3956c5465" (UID: "7dde93dc-0113-45c7-a961-e6d3956c5465"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:09:28.949646 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.949599    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dde93dc-0113-45c7-a961-e6d3956c5465-maas-api-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:09:28.949870 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.949856    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dde93dc-0113-45c7-a961-e6d3956c5465-metrics-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:09:28.949966 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.949953    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27xpc\" (UniqueName: \"kubernetes.io/projected/7dde93dc-0113-45c7-a961-e6d3956c5465-kube-api-access-27xpc\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:09:28.950070 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:28.950054    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dde93dc-0113-45c7-a961-e6d3956c5465-openshift-service-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:09:29.040688 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:29.040656    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7504140-a834-43d2-a00a-38d6fce32411" path="/var/lib/kubelet/pods/a7504140-a834-43d2-a00a-38d6fce32411/volumes"
Sep 15 19:09:29.598340 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:29.598308    2555 generic.go:356] "Generic (PLEG): container finished" podID="7dde93dc-0113-45c7-a961-e6d3956c5465" containerID="22a5fe0b12e8d237495ce527851a25a6b3f20fcaceacfd37e1f840eda613a76f" exitCode=0
Sep 15 19:09:29.598528 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:29.598355    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs" event={"ID":"7dde93dc-0113-45c7-a961-e6d3956c5465","Type":"ContainerDied","Data":"22a5fe0b12e8d237495ce527851a25a6b3f20fcaceacfd37e1f840eda613a76f"}
Sep 15 19:09:29.598528 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:29.598370    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"
Sep 15 19:09:29.598528 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:29.598396    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs" event={"ID":"7dde93dc-0113-45c7-a961-e6d3956c5465","Type":"ContainerDied","Data":"b7a423374ba17bb28e3be40eb5cc15bd613d2d0c84b908ce3a54e508eb67e50e"}
Sep 15 19:09:29.598528 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:29.598417    2555 scope.go:117] "RemoveContainer" containerID="22a5fe0b12e8d237495ce527851a25a6b3f20fcaceacfd37e1f840eda613a76f"
Sep 15 19:09:29.607844 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:29.607825    2555 scope.go:117] "RemoveContainer" containerID="22a5fe0b12e8d237495ce527851a25a6b3f20fcaceacfd37e1f840eda613a76f"
Sep 15 19:09:29.608113 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:09:29.608097    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22a5fe0b12e8d237495ce527851a25a6b3f20fcaceacfd37e1f840eda613a76f\": container with ID starting with 22a5fe0b12e8d237495ce527851a25a6b3f20fcaceacfd37e1f840eda613a76f not found: ID does not exist" containerID="22a5fe0b12e8d237495ce527851a25a6b3f20fcaceacfd37e1f840eda613a76f"
Sep 15 19:09:29.608162 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:29.608118    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22a5fe0b12e8d237495ce527851a25a6b3f20fcaceacfd37e1f840eda613a76f"} err="failed to get container status \"22a5fe0b12e8d237495ce527851a25a6b3f20fcaceacfd37e1f840eda613a76f\": rpc error: code = NotFound desc = could not find container \"22a5fe0b12e8d237495ce527851a25a6b3f20fcaceacfd37e1f840eda613a76f\": container with ID starting with 22a5fe0b12e8d237495ce527851a25a6b3f20fcaceacfd37e1f840eda613a76f not found: ID does not exist"
Sep 15 19:09:29.615411 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:29.615387    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"]
Sep 15 19:09:29.620910 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:29.620883    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84ee0c4c-7f9f5b76c7-m9bzs"]
Sep 15 19:09:31.040648 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:31.040598    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dde93dc-0113-45c7-a961-e6d3956c5465" path="/var/lib/kubelet/pods/7dde93dc-0113-45c7-a961-e6d3956c5465/volumes"
Sep 15 19:09:40.223473 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.223433    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"]
Sep 15 19:09:40.224118 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.224097    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc25a84c-4256-44f4-8f51-3cd1dc33cf80" containerName="maas-api"
Sep 15 19:09:40.224165 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.224124    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc25a84c-4256-44f4-8f51-3cd1dc33cf80" containerName="maas-api"
Sep 15 19:09:40.224165 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.224149    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7504140-a834-43d2-a00a-38d6fce32411" containerName="main"
Sep 15 19:09:40.224165 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.224159    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7504140-a834-43d2-a00a-38d6fce32411" containerName="main"
Sep 15 19:09:40.224263 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.224203    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dde93dc-0113-45c7-a961-e6d3956c5465" containerName="maas-api"
Sep 15 19:09:40.224263 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.224212    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dde93dc-0113-45c7-a961-e6d3956c5465" containerName="maas-api"
Sep 15 19:09:40.224329 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.224309    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7504140-a834-43d2-a00a-38d6fce32411" containerName="main"
Sep 15 19:09:40.224362 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.224330    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc25a84c-4256-44f4-8f51-3cd1dc33cf80" containerName="maas-api"
Sep 15 19:09:40.224362 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.224340    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dde93dc-0113-45c7-a961-e6d3956c5465" containerName="maas-api"
Sep 15 19:09:40.229433 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.229415    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.231731 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.231711    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-74ef1f\"/\"distinct-2-w0-74ef1f-kserve-self-signed-certs\""
Sep 15 19:09:40.236364 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.236342    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"]
Sep 15 19:09:40.358642 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.358575    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-tmp-dir\") pod \"distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.358841 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.358713    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-dshm\") pod \"distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.358841 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.358765    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-model-cache\") pod \"distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.358841 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.358804    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sz24\" (UniqueName: \"kubernetes.io/projected/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-kube-api-access-2sz24\") pod \"distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.358991 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.358873    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-home\") pod \"distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.358991 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.358918    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-tls-certs\") pod \"distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.459856 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.459824    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-tls-certs\") pod \"distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.460064 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.459949    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-tmp-dir\") pod \"distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.460064 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.460031    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-dshm\") pod \"distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.460064 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.460061    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-model-cache\") pod \"distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.460234 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.460084    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sz24\" (UniqueName: \"kubernetes.io/projected/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-kube-api-access-2sz24\") pod \"distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.460329 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.460309    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-home\") pod \"distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.460422 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.460396    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-tmp-dir\") pod \"distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.460622 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.460492    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-home\") pod \"distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.460723 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.460548    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-model-cache\") pod \"distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.462296 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.462271    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-dshm\") pod \"distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.462450 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.462435    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-tls-certs\") pod \"distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.469671 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.469643    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sz24\" (UniqueName: \"kubernetes.io/projected/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-kube-api-access-2sz24\") pod \"distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") " pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.540575 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.540540    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:40.814945 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:40.814916    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"]
Sep 15 19:09:40.816277 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:09:40.816254    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e2aaba7_d35b_48a9_9927_beb5a9a35ad9.slice/crio-670ff726097506825552c6ff9dfd1fed0a6faa805ced294c2771c415bb8807c4 WatchSource:0}: Error finding container 670ff726097506825552c6ff9dfd1fed0a6faa805ced294c2771c415bb8807c4: Status 404 returned error can't find the container with id 670ff726097506825552c6ff9dfd1fed0a6faa805ced294c2771c415bb8807c4
Sep 15 19:09:41.665705 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:41.665672    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn" event={"ID":"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9","Type":"ContainerStarted","Data":"a40f7f496b6233e2360a166f604d19c15afcc7765dd9ae15f7aafc3c67b1aede"}
Sep 15 19:09:41.665705 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:41.665712    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn" event={"ID":"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9","Type":"ContainerStarted","Data":"670ff726097506825552c6ff9dfd1fed0a6faa805ced294c2771c415bb8807c4"}
Sep 15 19:09:41.681455 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:41.681386    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn" podStartSLOduration=1.6813701779999999 podStartE2EDuration="1.681370178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:09:41.680858737 +0000 UTC m=+1123.168187722" watchObservedRunningTime="2026-09-15 19:09:41.681370178 +0000 UTC m=+1123.168699183"
Sep 15 19:09:44.664307 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:44.664266    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"]
Sep 15 19:09:44.664756 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:44.664524    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl" podUID="e34a51b1-98b4-49a7-914b-ab0949e57525" containerName="maas-api" containerID="cri-o://0ec7d30398e2aab87c27e303a9cd82a974e65ca99079ae14c82b9f9742c278f2" gracePeriod=30
Sep 15 19:09:44.918527 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:44.918462    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"
Sep 15 19:09:45.008811 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.008783    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e34a51b1-98b4-49a7-914b-ab0949e57525-openshift-service-ca\") pod \"e34a51b1-98b4-49a7-914b-ab0949e57525\" (UID: \"e34a51b1-98b4-49a7-914b-ab0949e57525\") "
Sep 15 19:09:45.009004 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.008838    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e34a51b1-98b4-49a7-914b-ab0949e57525-metrics-certs\") pod \"e34a51b1-98b4-49a7-914b-ab0949e57525\" (UID: \"e34a51b1-98b4-49a7-914b-ab0949e57525\") "
Sep 15 19:09:45.009004 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.008915    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqk99\" (UniqueName: \"kubernetes.io/projected/e34a51b1-98b4-49a7-914b-ab0949e57525-kube-api-access-rqk99\") pod \"e34a51b1-98b4-49a7-914b-ab0949e57525\" (UID: \"e34a51b1-98b4-49a7-914b-ab0949e57525\") "
Sep 15 19:09:45.009004 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.008990    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e34a51b1-98b4-49a7-914b-ab0949e57525-maas-api-tls\") pod \"e34a51b1-98b4-49a7-914b-ab0949e57525\" (UID: \"e34a51b1-98b4-49a7-914b-ab0949e57525\") "
Sep 15 19:09:45.009279 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.009248    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e34a51b1-98b4-49a7-914b-ab0949e57525-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e34a51b1-98b4-49a7-914b-ab0949e57525" (UID: "e34a51b1-98b4-49a7-914b-ab0949e57525"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:09:45.011121 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.011091    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e34a51b1-98b4-49a7-914b-ab0949e57525-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e34a51b1-98b4-49a7-914b-ab0949e57525" (UID: "e34a51b1-98b4-49a7-914b-ab0949e57525"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:09:45.011243 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.011136    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34a51b1-98b4-49a7-914b-ab0949e57525-kube-api-access-rqk99" (OuterVolumeSpecName: "kube-api-access-rqk99") pod "e34a51b1-98b4-49a7-914b-ab0949e57525" (UID: "e34a51b1-98b4-49a7-914b-ab0949e57525"). InnerVolumeSpecName "kube-api-access-rqk99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:45.011243 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.011191    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e34a51b1-98b4-49a7-914b-ab0949e57525-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e34a51b1-98b4-49a7-914b-ab0949e57525" (UID: "e34a51b1-98b4-49a7-914b-ab0949e57525"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:09:45.111525 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.111482    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e34a51b1-98b4-49a7-914b-ab0949e57525-metrics-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:09:45.111525 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.111514    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqk99\" (UniqueName: \"kubernetes.io/projected/e34a51b1-98b4-49a7-914b-ab0949e57525-kube-api-access-rqk99\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:09:45.111525 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.111530    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e34a51b1-98b4-49a7-914b-ab0949e57525-maas-api-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:09:45.111785 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.111629    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e34a51b1-98b4-49a7-914b-ab0949e57525-openshift-service-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:09:45.701436 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.701396    2555 generic.go:356] "Generic (PLEG): container finished" podID="e34a51b1-98b4-49a7-914b-ab0949e57525" containerID="0ec7d30398e2aab87c27e303a9cd82a974e65ca99079ae14c82b9f9742c278f2" exitCode=0
Sep 15 19:09:45.701867 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.701451    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"
Sep 15 19:09:45.701867 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.701481    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl" event={"ID":"e34a51b1-98b4-49a7-914b-ab0949e57525","Type":"ContainerDied","Data":"0ec7d30398e2aab87c27e303a9cd82a974e65ca99079ae14c82b9f9742c278f2"}
Sep 15 19:09:45.701867 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.701540    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl" event={"ID":"e34a51b1-98b4-49a7-914b-ab0949e57525","Type":"ContainerDied","Data":"67ffb3db277b954265986ee5af04dcb9b8ff1bf06d14ecabe47067c1c56e86a8"}
Sep 15 19:09:45.701867 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.701565    2555 scope.go:117] "RemoveContainer" containerID="0ec7d30398e2aab87c27e303a9cd82a974e65ca99079ae14c82b9f9742c278f2"
Sep 15 19:09:45.711583 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.711566    2555 scope.go:117] "RemoveContainer" containerID="0ec7d30398e2aab87c27e303a9cd82a974e65ca99079ae14c82b9f9742c278f2"
Sep 15 19:09:45.711902 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:09:45.711884    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec7d30398e2aab87c27e303a9cd82a974e65ca99079ae14c82b9f9742c278f2\": container with ID starting with 0ec7d30398e2aab87c27e303a9cd82a974e65ca99079ae14c82b9f9742c278f2 not found: ID does not exist" containerID="0ec7d30398e2aab87c27e303a9cd82a974e65ca99079ae14c82b9f9742c278f2"
Sep 15 19:09:45.711957 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.711913    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec7d30398e2aab87c27e303a9cd82a974e65ca99079ae14c82b9f9742c278f2"} err="failed to get container status \"0ec7d30398e2aab87c27e303a9cd82a974e65ca99079ae14c82b9f9742c278f2\": rpc error: code = NotFound desc = could not find container \"0ec7d30398e2aab87c27e303a9cd82a974e65ca99079ae14c82b9f9742c278f2\": container with ID starting with 0ec7d30398e2aab87c27e303a9cd82a974e65ca99079ae14c82b9f9742c278f2 not found: ID does not exist"
Sep 15 19:09:45.716345 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.716316    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"]
Sep 15 19:09:45.718251 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:45.718224    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08e9c4-855d7f45cf-fxvcl"]
Sep 15 19:09:47.040533 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:47.040498    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e34a51b1-98b4-49a7-914b-ab0949e57525" path="/var/lib/kubelet/pods/e34a51b1-98b4-49a7-914b-ab0949e57525/volumes"
Sep 15 19:09:50.541376 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:50.541330    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:50.541835 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:50.541417    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:50.554339 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:50.554302    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:50.742297 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:50.742265    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:09:54.378408 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.378368    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"]
Sep 15 19:09:54.379016 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.378998    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e34a51b1-98b4-49a7-914b-ab0949e57525" containerName="maas-api"
Sep 15 19:09:54.379166 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.379018    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e34a51b1-98b4-49a7-914b-ab0949e57525" containerName="maas-api"
Sep 15 19:09:54.379166 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.379151    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e34a51b1-98b4-49a7-914b-ab0949e57525" containerName="maas-api"
Sep 15 19:09:54.384036 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.384015    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"
Sep 15 19:09:54.386127 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.386110    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-4d299f-serving-cert\""
Sep 15 19:09:54.390014 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.389958    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"]
Sep 15 19:09:54.508281 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.508246    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz\" (UID: \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"
Sep 15 19:09:54.508476 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.508307    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qkq2\" (UniqueName: \"kubernetes.io/projected/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-kube-api-access-9qkq2\") pod \"maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz\" (UID: \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"
Sep 15 19:09:54.508476 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.508406    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz\" (UID: \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"
Sep 15 19:09:54.508578 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.508521    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz\" (UID: \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"
Sep 15 19:09:54.609393 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.609357    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz\" (UID: \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"
Sep 15 19:09:54.609572 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.609411    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz\" (UID: \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"
Sep 15 19:09:54.609572 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.609465    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qkq2\" (UniqueName: \"kubernetes.io/projected/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-kube-api-access-9qkq2\") pod \"maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz\" (UID: \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"
Sep 15 19:09:54.609572 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.609514    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz\" (UID: \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"
Sep 15 19:09:54.610215 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.610196    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz\" (UID: \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"
Sep 15 19:09:54.611826 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.611805    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz\" (UID: \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"
Sep 15 19:09:54.611943 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.611863    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz\" (UID: \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"
Sep 15 19:09:54.616431 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.616405    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qkq2\" (UniqueName: \"kubernetes.io/projected/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-kube-api-access-9qkq2\") pod \"maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz\" (UID: \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"
Sep 15 19:09:54.696737 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.696652    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"
Sep 15 19:09:54.971859 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:54.971832    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"]
Sep 15 19:09:54.973906 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:09:54.973876    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3216a3_8c92_4e1b_884a_bf5e90d1f772.slice/crio-2c47a2bbe1525acadc088acd68b56a8e1c9e5fe82e14c6d36da1f016068a422e WatchSource:0}: Error finding container 2c47a2bbe1525acadc088acd68b56a8e1c9e5fe82e14c6d36da1f016068a422e: Status 404 returned error can't find the container with id 2c47a2bbe1525acadc088acd68b56a8e1c9e5fe82e14c6d36da1f016068a422e
Sep 15 19:09:55.764286 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:55.764250    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz" event={"ID":"dd3216a3-8c92-4e1b-884a-bf5e90d1f772","Type":"ContainerStarted","Data":"7fb9c409ee1ea066d38a0157bb56769c68ce9ec81b0dc72213ef029305c207d1"}
Sep 15 19:09:55.764701 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:55.764310    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz" event={"ID":"dd3216a3-8c92-4e1b-884a-bf5e90d1f772","Type":"ContainerStarted","Data":"2c47a2bbe1525acadc088acd68b56a8e1c9e5fe82e14c6d36da1f016068a422e"}
Sep 15 19:09:55.764701 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:55.764337    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"
Sep 15 19:09:55.781845 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:55.781789    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz" podStartSLOduration=1.412504063 podStartE2EDuration="1.781774822s" totalImagesPullingTime="369.270759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:09:55.78002228 +0000 UTC m=+1137.267351265" watchObservedRunningTime="2026-09-15 19:09:55.781774822 +0000 UTC m=+1137.269103806"
Sep 15 19:09:58.239392 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:58.239360    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"]
Sep 15 19:09:58.244378 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:58.243809    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"
Sep 15 19:09:58.246157 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:58.246134    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-7073b8-serving-cert\""
Sep 15 19:09:58.251913 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:58.251885    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"]
Sep 15 19:09:58.347729 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:58.347701    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h\" (UID: \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"
Sep 15 19:09:58.347896 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:58.347737    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h\" (UID: \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"
Sep 15 19:09:58.347896 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:58.347825    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h\" (UID: \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"
Sep 15 19:09:58.347896 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:58.347848    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whzlv\" (UniqueName: \"kubernetes.io/projected/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-kube-api-access-whzlv\") pod \"maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h\" (UID: \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"
Sep 15 19:09:58.448878 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:58.448846    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h\" (UID: \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"
Sep 15 19:09:58.449042 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:58.448904    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h\" (UID: \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"
Sep 15 19:09:58.449042 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:58.449020    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h\" (UID: \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"
Sep 15 19:09:58.449140 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:58.449049    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whzlv\" (UniqueName: \"kubernetes.io/projected/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-kube-api-access-whzlv\") pod \"maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h\" (UID: \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"
Sep 15 19:09:58.449196 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:09:58.449163    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-serving-cert: secret "maas-api-e2e-shared-a-w2-7073b8-serving-cert" not found
Sep 15 19:09:58.449261 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:09:58.449250    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-maas-api-tls podName:33608ba7-ea93-48ff-b4d8-bbe0dab5ac13 nodeName:}" failed. No retries permitted until 2026-09-15 19:09:58.949231355 +0000 UTC m=+1140.436560336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-maas-api-tls") pod "maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h" (UID: "33608ba7-ea93-48ff-b4d8-bbe0dab5ac13") : secret "maas-api-e2e-shared-a-w2-7073b8-serving-cert" not found
Sep 15 19:09:58.449485 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:58.449466    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h\" (UID: \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"
Sep 15 19:09:58.451337 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:58.451305    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h\" (UID: \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"
Sep 15 19:09:58.457052 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:58.457030    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whzlv\" (UniqueName: \"kubernetes.io/projected/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-kube-api-access-whzlv\") pod \"maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h\" (UID: \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"
Sep 15 19:09:58.953562 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:58.953529    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h\" (UID: \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"
Sep 15 19:09:58.955903 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:58.955872    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h\" (UID: \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"
Sep 15 19:09:59.156341 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:59.156307    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"
Sep 15 19:09:59.425965 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:59.425939    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"]
Sep 15 19:09:59.428432 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:09:59.428401    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33608ba7_ea93_48ff_b4d8_bbe0dab5ac13.slice/crio-62f87a8da50f006399d60d01c32ddaa491676e4832eb1c482b4bb73155eda07d WatchSource:0}: Error finding container 62f87a8da50f006399d60d01c32ddaa491676e4832eb1c482b4bb73155eda07d: Status 404 returned error can't find the container with id 62f87a8da50f006399d60d01c32ddaa491676e4832eb1c482b4bb73155eda07d
Sep 15 19:09:59.793641 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:09:59.793590    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h" event={"ID":"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13","Type":"ContainerStarted","Data":"62f87a8da50f006399d60d01c32ddaa491676e4832eb1c482b4bb73155eda07d"}
Sep 15 19:10:00.800731 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:00.800689    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h" event={"ID":"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13","Type":"ContainerStarted","Data":"bb104cb7cf63546d6f48e96c31191ba6eafb6ac1a555aeb864a17ac012a0f462"}
Sep 15 19:10:00.801128 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:00.800785    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"
Sep 15 19:10:00.816339 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:00.816288    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h" podStartSLOduration=2.46318985 podStartE2EDuration="2.816274849s" totalImagesPullingTime="353.084999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:10:00.814691118 +0000 UTC m=+1142.302020104" watchObservedRunningTime="2026-09-15 19:10:00.816274849 +0000 UTC m=+1142.303603835"
Sep 15 19:10:01.775757 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:01.775728    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"
Sep 15 19:10:06.811733 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:06.811706    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"
Sep 15 19:10:25.649582 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:25.649546    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"]
Sep 15 19:10:25.654149 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:25.654126    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"
Sep 15 19:10:25.656323 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:25.656302    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-33ff04-serving-cert\""
Sep 15 19:10:25.662128 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:25.662100    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"]
Sep 15 19:10:25.733064 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:25.733028    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/563730fe-d2dd-4e64-a872-d4247466886e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd\" (UID: \"563730fe-d2dd-4e64-a872-d4247466886e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"
Sep 15 19:10:25.733238 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:25.733102    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563730fe-d2dd-4e64-a872-d4247466886e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd\" (UID: \"563730fe-d2dd-4e64-a872-d4247466886e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"
Sep 15 19:10:25.733238 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:25.733126    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsgxj\" (UniqueName: \"kubernetes.io/projected/563730fe-d2dd-4e64-a872-d4247466886e-kube-api-access-tsgxj\") pod \"maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd\" (UID: \"563730fe-d2dd-4e64-a872-d4247466886e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"
Sep 15 19:10:25.733238 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:25.733148    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563730fe-d2dd-4e64-a872-d4247466886e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd\" (UID: \"563730fe-d2dd-4e64-a872-d4247466886e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"
Sep 15 19:10:25.834100 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:25.834061    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563730fe-d2dd-4e64-a872-d4247466886e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd\" (UID: \"563730fe-d2dd-4e64-a872-d4247466886e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"
Sep 15 19:10:25.834100 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:25.834098    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsgxj\" (UniqueName: \"kubernetes.io/projected/563730fe-d2dd-4e64-a872-d4247466886e-kube-api-access-tsgxj\") pod \"maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd\" (UID: \"563730fe-d2dd-4e64-a872-d4247466886e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"
Sep 15 19:10:25.834331 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:25.834120    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563730fe-d2dd-4e64-a872-d4247466886e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd\" (UID: \"563730fe-d2dd-4e64-a872-d4247466886e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"
Sep 15 19:10:25.834331 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:25.834189    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/563730fe-d2dd-4e64-a872-d4247466886e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd\" (UID: \"563730fe-d2dd-4e64-a872-d4247466886e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"
Sep 15 19:10:25.834331 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:10:25.834217    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-serving-cert: secret "maas-api-e2e-shared-b-w2-33ff04-serving-cert" not found
Sep 15 19:10:25.834331 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:10:25.834283    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/563730fe-d2dd-4e64-a872-d4247466886e-maas-api-tls podName:563730fe-d2dd-4e64-a872-d4247466886e nodeName:}" failed. No retries permitted until 2026-09-15 19:10:26.334265245 +0000 UTC m=+1167.821594225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/563730fe-d2dd-4e64-a872-d4247466886e-maas-api-tls") pod "maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd" (UID: "563730fe-d2dd-4e64-a872-d4247466886e") : secret "maas-api-e2e-shared-b-w2-33ff04-serving-cert" not found
Sep 15 19:10:25.834882 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:25.834861    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563730fe-d2dd-4e64-a872-d4247466886e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd\" (UID: \"563730fe-d2dd-4e64-a872-d4247466886e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"
Sep 15 19:10:25.836701 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:25.836683    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/563730fe-d2dd-4e64-a872-d4247466886e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd\" (UID: \"563730fe-d2dd-4e64-a872-d4247466886e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"
Sep 15 19:10:25.841908 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:25.841886    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsgxj\" (UniqueName: \"kubernetes.io/projected/563730fe-d2dd-4e64-a872-d4247466886e-kube-api-access-tsgxj\") pod \"maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd\" (UID: \"563730fe-d2dd-4e64-a872-d4247466886e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"
Sep 15 19:10:26.339331 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:26.339301    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563730fe-d2dd-4e64-a872-d4247466886e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd\" (UID: \"563730fe-d2dd-4e64-a872-d4247466886e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"
Sep 15 19:10:26.341803 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:26.341774    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563730fe-d2dd-4e64-a872-d4247466886e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd\" (UID: \"563730fe-d2dd-4e64-a872-d4247466886e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"
Sep 15 19:10:26.567821 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:26.567789    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"
Sep 15 19:10:26.839853 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:26.839823    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"]
Sep 15 19:10:26.841913 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:10:26.841885    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563730fe_d2dd_4e64_a872_d4247466886e.slice/crio-c4feb95dbc39a234fda92f93d4efbd41772f2bda6c34d78beca3ff3a23780900 WatchSource:0}: Error finding container c4feb95dbc39a234fda92f93d4efbd41772f2bda6c34d78beca3ff3a23780900: Status 404 returned error can't find the container with id c4feb95dbc39a234fda92f93d4efbd41772f2bda6c34d78beca3ff3a23780900
Sep 15 19:10:26.957083 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:26.956990    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd" event={"ID":"563730fe-d2dd-4e64-a872-d4247466886e","Type":"ContainerStarted","Data":"c4feb95dbc39a234fda92f93d4efbd41772f2bda6c34d78beca3ff3a23780900"}
Sep 15 19:10:27.964553 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:27.964518    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd" event={"ID":"563730fe-d2dd-4e64-a872-d4247466886e","Type":"ContainerStarted","Data":"fa941bfc790180ae2a1626f1fdad221a5ad38565131956c6841d5ede13b25448"}
Sep 15 19:10:27.965044 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:27.964707    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"
Sep 15 19:10:27.982963 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:27.982863    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd" podStartSLOduration=2.096457176 podStartE2EDuration="2.982845267s" totalImagesPullingTime="886.388091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:10:27.98191438 +0000 UTC m=+1169.469243365" watchObservedRunningTime="2026-09-15 19:10:27.982845267 +0000 UTC m=+1169.470174251"
Sep 15 19:10:28.164987 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:28.164950    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"]
Sep 15 19:10:28.168700 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:28.168680    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"
Sep 15 19:10:28.170919 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:28.170890    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-501bed-serving-cert\""
Sep 15 19:10:28.179544 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:28.179521    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"]
Sep 15 19:10:28.261446 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:28.261416    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08b3386-95ee-42ab-8fd6-5787bed9336a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s\" (UID: \"d08b3386-95ee-42ab-8fd6-5787bed9336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"
Sep 15 19:10:28.261622 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:28.261466    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08b3386-95ee-42ab-8fd6-5787bed9336a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s\" (UID: \"d08b3386-95ee-42ab-8fd6-5787bed9336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"
Sep 15 19:10:28.261622 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:28.261487    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08b3386-95ee-42ab-8fd6-5787bed9336a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s\" (UID: \"d08b3386-95ee-42ab-8fd6-5787bed9336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"
Sep 15 19:10:28.261622 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:28.261558    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27kq8\" (UniqueName: \"kubernetes.io/projected/d08b3386-95ee-42ab-8fd6-5787bed9336a-kube-api-access-27kq8\") pod \"maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s\" (UID: \"d08b3386-95ee-42ab-8fd6-5787bed9336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"
Sep 15 19:10:28.362383 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:28.362351    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08b3386-95ee-42ab-8fd6-5787bed9336a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s\" (UID: \"d08b3386-95ee-42ab-8fd6-5787bed9336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"
Sep 15 19:10:28.362572 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:28.362400    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08b3386-95ee-42ab-8fd6-5787bed9336a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s\" (UID: \"d08b3386-95ee-42ab-8fd6-5787bed9336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"
Sep 15 19:10:28.362572 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:28.362417    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08b3386-95ee-42ab-8fd6-5787bed9336a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s\" (UID: \"d08b3386-95ee-42ab-8fd6-5787bed9336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"
Sep 15 19:10:28.362572 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:28.362463    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27kq8\" (UniqueName: \"kubernetes.io/projected/d08b3386-95ee-42ab-8fd6-5787bed9336a-kube-api-access-27kq8\") pod \"maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s\" (UID: \"d08b3386-95ee-42ab-8fd6-5787bed9336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"
Sep 15 19:10:28.363135 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:28.363109    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08b3386-95ee-42ab-8fd6-5787bed9336a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s\" (UID: \"d08b3386-95ee-42ab-8fd6-5787bed9336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"
Sep 15 19:10:28.364981 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:28.364958    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08b3386-95ee-42ab-8fd6-5787bed9336a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s\" (UID: \"d08b3386-95ee-42ab-8fd6-5787bed9336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"
Sep 15 19:10:28.365075 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:28.364958    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08b3386-95ee-42ab-8fd6-5787bed9336a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s\" (UID: \"d08b3386-95ee-42ab-8fd6-5787bed9336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"
Sep 15 19:10:28.370835 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:28.370812    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27kq8\" (UniqueName: \"kubernetes.io/projected/d08b3386-95ee-42ab-8fd6-5787bed9336a-kube-api-access-27kq8\") pod \"maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s\" (UID: \"d08b3386-95ee-42ab-8fd6-5787bed9336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"
Sep 15 19:10:28.481252 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:28.481216    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"
Sep 15 19:10:28.757065 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:28.756795    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"]
Sep 15 19:10:28.759915 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:10:28.759883    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08b3386_95ee_42ab_8fd6_5787bed9336a.slice/crio-6c0dca8f638f8a7f600962e130d48542a4cea6779ad0bd542c1331d4b4c3b832 WatchSource:0}: Error finding container 6c0dca8f638f8a7f600962e130d48542a4cea6779ad0bd542c1331d4b4c3b832: Status 404 returned error can't find the container with id 6c0dca8f638f8a7f600962e130d48542a4cea6779ad0bd542c1331d4b4c3b832
Sep 15 19:10:28.971246 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:28.971159    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s" event={"ID":"d08b3386-95ee-42ab-8fd6-5787bed9336a","Type":"ContainerStarted","Data":"6c0dca8f638f8a7f600962e130d48542a4cea6779ad0bd542c1331d4b4c3b832"}
Sep 15 19:10:29.979384 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:29.979351    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s" event={"ID":"d08b3386-95ee-42ab-8fd6-5787bed9336a","Type":"ContainerStarted","Data":"6b197ace1acd653f51e362fafc7ac3657f474cefcb138434177076f576b71887"}
Sep 15 19:10:29.979804 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:29.979449    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"
Sep 15 19:10:30.009519 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:30.009458    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s" podStartSLOduration=1.6090730020000001 podStartE2EDuration="2.00944346s" totalImagesPullingTime="400.370458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:10:30.008258856 +0000 UTC m=+1171.495587841" watchObservedRunningTime="2026-09-15 19:10:30.00944346 +0000 UTC m=+1171.496772445"
Sep 15 19:10:33.977209 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:33.977181    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"
Sep 15 19:10:35.992281 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:35.992252    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"
Sep 15 19:10:52.859023 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:52.858966    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8924-e60f95"]
Sep 15 19:10:52.863908 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:52.863888    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8924-e60f95"
Sep 15 19:10:52.866677 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:52.866641    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zrrnn\""
Sep 15 19:10:52.869122 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:52.869091    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8924-e60f95"]
Sep 15 19:10:52.908983 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:52.908943    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff9qn\" (UniqueName: \"kubernetes.io/projected/37925b7a-406e-4e03-b49d-852de50c995f-kube-api-access-ff9qn\") pod \"test-curl-8924-e60f95\" (UID: \"37925b7a-406e-4e03-b49d-852de50c995f\") " pod="opendatahub/test-curl-8924-e60f95"
Sep 15 19:10:53.010106 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:53.010071    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff9qn\" (UniqueName: \"kubernetes.io/projected/37925b7a-406e-4e03-b49d-852de50c995f-kube-api-access-ff9qn\") pod \"test-curl-8924-e60f95\" (UID: \"37925b7a-406e-4e03-b49d-852de50c995f\") " pod="opendatahub/test-curl-8924-e60f95"
Sep 15 19:10:53.018547 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:53.018524    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff9qn\" (UniqueName: \"kubernetes.io/projected/37925b7a-406e-4e03-b49d-852de50c995f-kube-api-access-ff9qn\") pod \"test-curl-8924-e60f95\" (UID: \"37925b7a-406e-4e03-b49d-852de50c995f\") " pod="opendatahub/test-curl-8924-e60f95"
Sep 15 19:10:53.175769 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:53.175679    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8924-e60f95"
Sep 15 19:10:53.446369 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:53.446338    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8924-e60f95"]
Sep 15 19:10:53.448884 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:10:53.448853    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37925b7a_406e_4e03_b49d_852de50c995f.slice/crio-22235ba6924560f99450bd66d9471b6c966e0b7bf7dd8b533adb8521f548bb2a WatchSource:0}: Error finding container 22235ba6924560f99450bd66d9471b6c966e0b7bf7dd8b533adb8521f548bb2a: Status 404 returned error can't find the container with id 22235ba6924560f99450bd66d9471b6c966e0b7bf7dd8b533adb8521f548bb2a
Sep 15 19:10:54.115771 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:54.115737    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8924-e60f95" event={"ID":"37925b7a-406e-4e03-b49d-852de50c995f","Type":"ContainerStarted","Data":"9b7afd1bcc71fdd533f33317c99d313e7676a239f92017c7a180d4623c3495bb"}
Sep 15 19:10:54.116168 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:54.115781    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8924-e60f95" event={"ID":"37925b7a-406e-4e03-b49d-852de50c995f","Type":"ContainerStarted","Data":"22235ba6924560f99450bd66d9471b6c966e0b7bf7dd8b533adb8521f548bb2a"}
Sep 15 19:10:54.130361 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:54.130305    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8924-e60f95" podStartSLOduration=2.130285636 podStartE2EDuration="2.130285636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:10:54.128202315 +0000 UTC m=+1195.615531301" watchObservedRunningTime="2026-09-15 19:10:54.130285636 +0000 UTC m=+1195.617614620"
Sep 15 19:10:54.720764 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:54.720729    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8924-e60f95"]
Sep 15 19:10:54.723520 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:54.723485    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8924-e60f95"]
Sep 15 19:10:55.121684 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:10:55.121648    2555 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8924-e60f95" container="test-curl-8924-e60f95"
Sep 15 19:10:56.127049 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:56.127003    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8924-e60f95" podUID="37925b7a-406e-4e03-b49d-852de50c995f" containerName="test-curl-8924-e60f95" containerID="cri-o://9b7afd1bcc71fdd533f33317c99d313e7676a239f92017c7a180d4623c3495bb" gracePeriod=2
Sep 15 19:10:58.307499 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:58.307472    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8924-e60f95"
Sep 15 19:10:58.367765 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:58.367727    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ff9qn\" (UniqueName: \"kubernetes.io/projected/37925b7a-406e-4e03-b49d-852de50c995f-kube-api-access-ff9qn\") pod \"37925b7a-406e-4e03-b49d-852de50c995f\" (UID: \"37925b7a-406e-4e03-b49d-852de50c995f\") "
Sep 15 19:10:58.369743 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:58.369721    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37925b7a-406e-4e03-b49d-852de50c995f-kube-api-access-ff9qn" (OuterVolumeSpecName: "kube-api-access-ff9qn") pod "37925b7a-406e-4e03-b49d-852de50c995f" (UID: "37925b7a-406e-4e03-b49d-852de50c995f"). InnerVolumeSpecName "kube-api-access-ff9qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:10:58.469233 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:58.469130    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff9qn\" (UniqueName: \"kubernetes.io/projected/37925b7a-406e-4e03-b49d-852de50c995f-kube-api-access-ff9qn\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:10:58.614019 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:58.613983    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8924-5e2166"]
Sep 15 19:10:58.614440 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:58.614427    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37925b7a-406e-4e03-b49d-852de50c995f" containerName="test-curl-8924-e60f95"
Sep 15 19:10:58.614490 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:58.614442    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="37925b7a-406e-4e03-b49d-852de50c995f" containerName="test-curl-8924-e60f95"
Sep 15 19:10:58.614525 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:58.614512    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="37925b7a-406e-4e03-b49d-852de50c995f" containerName="test-curl-8924-e60f95"
Sep 15 19:10:58.617682 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:58.617666    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8924-5e2166"
Sep 15 19:10:58.624158 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:58.624123    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8924-5e2166"]
Sep 15 19:10:58.773114 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:58.773078    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trd2t\" (UniqueName: \"kubernetes.io/projected/c45ac9e1-a299-45bc-b5b9-83e06f868032-kube-api-access-trd2t\") pod \"test-curl-8924-5e2166\" (UID: \"c45ac9e1-a299-45bc-b5b9-83e06f868032\") " pod="opendatahub/test-curl-8924-5e2166"
Sep 15 19:10:58.874281 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:58.874248    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trd2t\" (UniqueName: \"kubernetes.io/projected/c45ac9e1-a299-45bc-b5b9-83e06f868032-kube-api-access-trd2t\") pod \"test-curl-8924-5e2166\" (UID: \"c45ac9e1-a299-45bc-b5b9-83e06f868032\") " pod="opendatahub/test-curl-8924-5e2166"
Sep 15 19:10:58.882402 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:58.882375    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trd2t\" (UniqueName: \"kubernetes.io/projected/c45ac9e1-a299-45bc-b5b9-83e06f868032-kube-api-access-trd2t\") pod \"test-curl-8924-5e2166\" (UID: \"c45ac9e1-a299-45bc-b5b9-83e06f868032\") " pod="opendatahub/test-curl-8924-5e2166"
Sep 15 19:10:58.929696 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:58.929665    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8924-5e2166"
Sep 15 19:10:59.042271 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:59.042192    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37925b7a-406e-4e03-b49d-852de50c995f" path="/var/lib/kubelet/pods/37925b7a-406e-4e03-b49d-852de50c995f/volumes"
Sep 15 19:10:59.144726 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:59.144694    2555 generic.go:356] "Generic (PLEG): container finished" podID="37925b7a-406e-4e03-b49d-852de50c995f" containerID="9b7afd1bcc71fdd533f33317c99d313e7676a239f92017c7a180d4623c3495bb" exitCode=137
Sep 15 19:10:59.144912 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:59.144736    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8924-e60f95"
Sep 15 19:10:59.144912 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:59.144799    2555 scope.go:117] "RemoveContainer" containerID="9b7afd1bcc71fdd533f33317c99d313e7676a239f92017c7a180d4623c3495bb"
Sep 15 19:10:59.153321 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:59.153270    2555 scope.go:117] "RemoveContainer" containerID="9b7afd1bcc71fdd533f33317c99d313e7676a239f92017c7a180d4623c3495bb"
Sep 15 19:10:59.153585 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:10:59.153564    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7afd1bcc71fdd533f33317c99d313e7676a239f92017c7a180d4623c3495bb\": container with ID starting with 9b7afd1bcc71fdd533f33317c99d313e7676a239f92017c7a180d4623c3495bb not found: ID does not exist" containerID="9b7afd1bcc71fdd533f33317c99d313e7676a239f92017c7a180d4623c3495bb"
Sep 15 19:10:59.153740 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:59.153596    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7afd1bcc71fdd533f33317c99d313e7676a239f92017c7a180d4623c3495bb"} err="failed to get container status \"9b7afd1bcc71fdd533f33317c99d313e7676a239f92017c7a180d4623c3495bb\": rpc error: code = NotFound desc = could not find container \"9b7afd1bcc71fdd533f33317c99d313e7676a239f92017c7a180d4623c3495bb\": container with ID starting with 9b7afd1bcc71fdd533f33317c99d313e7676a239f92017c7a180d4623c3495bb not found: ID does not exist"
Sep 15 19:10:59.200206 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:10:59.200179    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8924-5e2166"]
Sep 15 19:10:59.201934 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:10:59.201906    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc45ac9e1_a299_45bc_b5b9_83e06f868032.slice/crio-ff40e77f1d923f9f7bb236dc48bec9b69d4e5c4ff90f1c69787eb3c235a37c61 WatchSource:0}: Error finding container ff40e77f1d923f9f7bb236dc48bec9b69d4e5c4ff90f1c69787eb3c235a37c61: Status 404 returned error can't find the container with id ff40e77f1d923f9f7bb236dc48bec9b69d4e5c4ff90f1c69787eb3c235a37c61
Sep 15 19:11:00.151467 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:00.151419    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8924-5e2166" event={"ID":"c45ac9e1-a299-45bc-b5b9-83e06f868032","Type":"ContainerStarted","Data":"29c2f4f760fd41b39214217fcf2299fac1305f46157627d01fc06b30bb37bbfe"}
Sep 15 19:11:00.151954 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:00.151480    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8924-5e2166" event={"ID":"c45ac9e1-a299-45bc-b5b9-83e06f868032","Type":"ContainerStarted","Data":"ff40e77f1d923f9f7bb236dc48bec9b69d4e5c4ff90f1c69787eb3c235a37c61"}
Sep 15 19:11:00.166582 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:00.166535    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8924-5e2166" podStartSLOduration=2.166520631 podStartE2EDuration="2.166520631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:11:00.164192728 +0000 UTC m=+1201.651521713" watchObservedRunningTime="2026-09-15 19:11:00.166520631 +0000 UTC m=+1201.653849662"
Sep 15 19:11:00.724974 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:00.724942    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8924-5e2166"]
Sep 15 19:11:00.729004 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:00.728972    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8924-5e2166"]
Sep 15 19:11:00.843500 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:00.843415    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8924-ca59be"]
Sep 15 19:11:00.843895 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:00.843881    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c45ac9e1-a299-45bc-b5b9-83e06f868032" containerName="test-curl-8924-5e2166"
Sep 15 19:11:00.843945 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:00.843897    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c45ac9e1-a299-45bc-b5b9-83e06f868032" containerName="test-curl-8924-5e2166"
Sep 15 19:11:00.843999 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:00.843990    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c45ac9e1-a299-45bc-b5b9-83e06f868032" containerName="test-curl-8924-5e2166"
Sep 15 19:11:00.847230 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:00.847213    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8924-ca59be"
Sep 15 19:11:00.852708 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:00.852682    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8924-ca59be"]
Sep 15 19:11:00.997176 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:00.997139    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbkd5\" (UniqueName: \"kubernetes.io/projected/d767bc47-9eb9-46b1-afbb-cd105bbb1b46-kube-api-access-hbkd5\") pod \"test-curl-8924-ca59be\" (UID: \"d767bc47-9eb9-46b1-afbb-cd105bbb1b46\") " pod="opendatahub/test-curl-8924-ca59be"
Sep 15 19:11:01.098331 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:01.098241    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbkd5\" (UniqueName: \"kubernetes.io/projected/d767bc47-9eb9-46b1-afbb-cd105bbb1b46-kube-api-access-hbkd5\") pod \"test-curl-8924-ca59be\" (UID: \"d767bc47-9eb9-46b1-afbb-cd105bbb1b46\") " pod="opendatahub/test-curl-8924-ca59be"
Sep 15 19:11:01.105655 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:01.105603    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbkd5\" (UniqueName: \"kubernetes.io/projected/d767bc47-9eb9-46b1-afbb-cd105bbb1b46-kube-api-access-hbkd5\") pod \"test-curl-8924-ca59be\" (UID: \"d767bc47-9eb9-46b1-afbb-cd105bbb1b46\") " pod="opendatahub/test-curl-8924-ca59be"
Sep 15 19:11:01.159341 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:01.159304    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8924-ca59be"
Sep 15 19:11:01.160939 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:11:01.160917    2555 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8924-5e2166" container="test-curl-8924-5e2166"
Sep 15 19:11:01.427351 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:01.427253    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8924-ca59be"]
Sep 15 19:11:01.429593 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:11:01.429568    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd767bc47_9eb9_46b1_afbb_cd105bbb1b46.slice/crio-d1b910564b316030c8e6352496401627fae37a64351713e6523f1915e2bbaa32 WatchSource:0}: Error finding container d1b910564b316030c8e6352496401627fae37a64351713e6523f1915e2bbaa32: Status 404 returned error can't find the container with id d1b910564b316030c8e6352496401627fae37a64351713e6523f1915e2bbaa32
Sep 15 19:11:02.169262 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:02.169219    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8924-ca59be" event={"ID":"d767bc47-9eb9-46b1-afbb-cd105bbb1b46","Type":"ContainerStarted","Data":"39a3365eacf15b5031093bc1f747b769b7b1a3dd3f05f31716695172ef21e154"}
Sep 15 19:11:02.169741 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:02.169274    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8924-ca59be" event={"ID":"d767bc47-9eb9-46b1-afbb-cd105bbb1b46","Type":"ContainerStarted","Data":"d1b910564b316030c8e6352496401627fae37a64351713e6523f1915e2bbaa32"}
Sep 15 19:11:02.169741 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:02.169384    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8924-5e2166" podUID="c45ac9e1-a299-45bc-b5b9-83e06f868032" containerName="test-curl-8924-5e2166" containerID="cri-o://29c2f4f760fd41b39214217fcf2299fac1305f46157627d01fc06b30bb37bbfe" gracePeriod=2
Sep 15 19:11:02.182347 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:02.182302    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8924-ca59be" podStartSLOduration=2.182287833 podStartE2EDuration="2.182287833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:11:02.181299205 +0000 UTC m=+1203.668628190" watchObservedRunningTime="2026-09-15 19:11:02.182287833 +0000 UTC m=+1203.669616818"
Sep 15 19:11:02.751806 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:02.751753    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8924-ca59be"]
Sep 15 19:11:02.755699 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:02.755668    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8924-ca59be"]
Sep 15 19:11:03.175412 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:11:03.175387    2555 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8924-ca59be" container="test-curl-8924-ca59be"
Sep 15 19:11:04.181702 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:04.181634    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8924-ca59be" podUID="d767bc47-9eb9-46b1-afbb-cd105bbb1b46" containerName="test-curl-8924-ca59be" containerID="cri-o://39a3365eacf15b5031093bc1f747b769b7b1a3dd3f05f31716695172ef21e154" gracePeriod=2
Sep 15 19:11:04.319116 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:04.319090    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8924-5e2166"
Sep 15 19:11:04.433385 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:04.433285    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trd2t\" (UniqueName: \"kubernetes.io/projected/c45ac9e1-a299-45bc-b5b9-83e06f868032-kube-api-access-trd2t\") pod \"c45ac9e1-a299-45bc-b5b9-83e06f868032\" (UID: \"c45ac9e1-a299-45bc-b5b9-83e06f868032\") "
Sep 15 19:11:04.435391 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:04.435363    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45ac9e1-a299-45bc-b5b9-83e06f868032-kube-api-access-trd2t" (OuterVolumeSpecName: "kube-api-access-trd2t") pod "c45ac9e1-a299-45bc-b5b9-83e06f868032" (UID: "c45ac9e1-a299-45bc-b5b9-83e06f868032"). InnerVolumeSpecName "kube-api-access-trd2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:11:04.534573 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:04.534540    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trd2t\" (UniqueName: \"kubernetes.io/projected/c45ac9e1-a299-45bc-b5b9-83e06f868032-kube-api-access-trd2t\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:05.041386 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:05.041349    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c45ac9e1-a299-45bc-b5b9-83e06f868032" path="/var/lib/kubelet/pods/c45ac9e1-a299-45bc-b5b9-83e06f868032/volumes"
Sep 15 19:11:05.189304 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:05.189269    2555 generic.go:356] "Generic (PLEG): container finished" podID="c45ac9e1-a299-45bc-b5b9-83e06f868032" containerID="29c2f4f760fd41b39214217fcf2299fac1305f46157627d01fc06b30bb37bbfe" exitCode=137
Sep 15 19:11:05.189710 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:05.189314    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8924-5e2166"
Sep 15 19:11:05.189710 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:05.189365    2555 scope.go:117] "RemoveContainer" containerID="29c2f4f760fd41b39214217fcf2299fac1305f46157627d01fc06b30bb37bbfe"
Sep 15 19:11:05.197882 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:05.197865    2555 scope.go:117] "RemoveContainer" containerID="29c2f4f760fd41b39214217fcf2299fac1305f46157627d01fc06b30bb37bbfe"
Sep 15 19:11:05.198136 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:11:05.198117    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29c2f4f760fd41b39214217fcf2299fac1305f46157627d01fc06b30bb37bbfe\": container with ID starting with 29c2f4f760fd41b39214217fcf2299fac1305f46157627d01fc06b30bb37bbfe not found: ID does not exist" containerID="29c2f4f760fd41b39214217fcf2299fac1305f46157627d01fc06b30bb37bbfe"
Sep 15 19:11:05.198193 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:05.198148    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29c2f4f760fd41b39214217fcf2299fac1305f46157627d01fc06b30bb37bbfe"} err="failed to get container status \"29c2f4f760fd41b39214217fcf2299fac1305f46157627d01fc06b30bb37bbfe\": rpc error: code = NotFound desc = could not find container \"29c2f4f760fd41b39214217fcf2299fac1305f46157627d01fc06b30bb37bbfe\": container with ID starting with 29c2f4f760fd41b39214217fcf2299fac1305f46157627d01fc06b30bb37bbfe not found: ID does not exist"
Sep 15 19:11:06.328851 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:06.328827    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8924-ca59be"
Sep 15 19:11:06.454349 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:06.454262    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbkd5\" (UniqueName: \"kubernetes.io/projected/d767bc47-9eb9-46b1-afbb-cd105bbb1b46-kube-api-access-hbkd5\") pod \"d767bc47-9eb9-46b1-afbb-cd105bbb1b46\" (UID: \"d767bc47-9eb9-46b1-afbb-cd105bbb1b46\") "
Sep 15 19:11:06.456675 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:06.456642    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d767bc47-9eb9-46b1-afbb-cd105bbb1b46-kube-api-access-hbkd5" (OuterVolumeSpecName: "kube-api-access-hbkd5") pod "d767bc47-9eb9-46b1-afbb-cd105bbb1b46" (UID: "d767bc47-9eb9-46b1-afbb-cd105bbb1b46"). InnerVolumeSpecName "kube-api-access-hbkd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:11:06.555328 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:06.555295    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbkd5\" (UniqueName: \"kubernetes.io/projected/d767bc47-9eb9-46b1-afbb-cd105bbb1b46-kube-api-access-hbkd5\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:07.040787 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:07.040758    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d767bc47-9eb9-46b1-afbb-cd105bbb1b46" path="/var/lib/kubelet/pods/d767bc47-9eb9-46b1-afbb-cd105bbb1b46/volumes"
Sep 15 19:11:07.203034 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:07.202997    2555 generic.go:356] "Generic (PLEG): container finished" podID="d767bc47-9eb9-46b1-afbb-cd105bbb1b46" containerID="39a3365eacf15b5031093bc1f747b769b7b1a3dd3f05f31716695172ef21e154" exitCode=137
Sep 15 19:11:07.203214 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:07.203040    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8924-ca59be"
Sep 15 19:11:07.203214 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:07.203051    2555 scope.go:117] "RemoveContainer" containerID="39a3365eacf15b5031093bc1f747b769b7b1a3dd3f05f31716695172ef21e154"
Sep 15 19:11:07.212114 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:07.212090    2555 scope.go:117] "RemoveContainer" containerID="39a3365eacf15b5031093bc1f747b769b7b1a3dd3f05f31716695172ef21e154"
Sep 15 19:11:07.212384 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:11:07.212364    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a3365eacf15b5031093bc1f747b769b7b1a3dd3f05f31716695172ef21e154\": container with ID starting with 39a3365eacf15b5031093bc1f747b769b7b1a3dd3f05f31716695172ef21e154 not found: ID does not exist" containerID="39a3365eacf15b5031093bc1f747b769b7b1a3dd3f05f31716695172ef21e154"
Sep 15 19:11:07.212437 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:07.212391    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a3365eacf15b5031093bc1f747b769b7b1a3dd3f05f31716695172ef21e154"} err="failed to get container status \"39a3365eacf15b5031093bc1f747b769b7b1a3dd3f05f31716695172ef21e154\": rpc error: code = NotFound desc = could not find container \"39a3365eacf15b5031093bc1f747b769b7b1a3dd3f05f31716695172ef21e154\": container with ID starting with 39a3365eacf15b5031093bc1f747b769b7b1a3dd3f05f31716695172ef21e154 not found: ID does not exist"
Sep 15 19:11:07.589152 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:07.589122    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"]
Sep 15 19:11:07.601955 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:07.601929    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"]
Sep 15 19:11:07.614529 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:07.614500    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"]
Sep 15 19:11:09.654693 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.654649    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"]
Sep 15 19:11:09.655182 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.655164    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d767bc47-9eb9-46b1-afbb-cd105bbb1b46" containerName="test-curl-8924-ca59be"
Sep 15 19:11:09.655259 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.655185    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d767bc47-9eb9-46b1-afbb-cd105bbb1b46" containerName="test-curl-8924-ca59be"
Sep 15 19:11:09.655354 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.655327    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d767bc47-9eb9-46b1-afbb-cd105bbb1b46" containerName="test-curl-8924-ca59be"
Sep 15 19:11:09.660183 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.660161    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"
Sep 15 19:11:09.662501 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.662482    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d2e6cc70-serving-cert\""
Sep 15 19:11:09.670664 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.670628    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"]
Sep 15 19:11:09.788703 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.788667    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64263691-60bb-4559-86ad-0ec935f6357a-openshift-service-ca\") pod \"maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh\" (UID: \"64263691-60bb-4559-86ad-0ec935f6357a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"
Sep 15 19:11:09.788903 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.788763    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbvwh\" (UniqueName: \"kubernetes.io/projected/64263691-60bb-4559-86ad-0ec935f6357a-kube-api-access-vbvwh\") pod \"maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh\" (UID: \"64263691-60bb-4559-86ad-0ec935f6357a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"
Sep 15 19:11:09.788903 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.788802    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64263691-60bb-4559-86ad-0ec935f6357a-metrics-certs\") pod \"maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh\" (UID: \"64263691-60bb-4559-86ad-0ec935f6357a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"
Sep 15 19:11:09.788903 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.788827    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64263691-60bb-4559-86ad-0ec935f6357a-maas-api-tls\") pod \"maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh\" (UID: \"64263691-60bb-4559-86ad-0ec935f6357a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"
Sep 15 19:11:09.890300 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.890268    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbvwh\" (UniqueName: \"kubernetes.io/projected/64263691-60bb-4559-86ad-0ec935f6357a-kube-api-access-vbvwh\") pod \"maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh\" (UID: \"64263691-60bb-4559-86ad-0ec935f6357a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"
Sep 15 19:11:09.890463 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.890322    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64263691-60bb-4559-86ad-0ec935f6357a-metrics-certs\") pod \"maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh\" (UID: \"64263691-60bb-4559-86ad-0ec935f6357a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"
Sep 15 19:11:09.890463 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.890365    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64263691-60bb-4559-86ad-0ec935f6357a-maas-api-tls\") pod \"maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh\" (UID: \"64263691-60bb-4559-86ad-0ec935f6357a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"
Sep 15 19:11:09.890463 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.890416    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64263691-60bb-4559-86ad-0ec935f6357a-openshift-service-ca\") pod \"maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh\" (UID: \"64263691-60bb-4559-86ad-0ec935f6357a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"
Sep 15 19:11:09.891114 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.891088    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64263691-60bb-4559-86ad-0ec935f6357a-openshift-service-ca\") pod \"maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh\" (UID: \"64263691-60bb-4559-86ad-0ec935f6357a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"
Sep 15 19:11:09.892795 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.892774    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64263691-60bb-4559-86ad-0ec935f6357a-metrics-certs\") pod \"maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh\" (UID: \"64263691-60bb-4559-86ad-0ec935f6357a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"
Sep 15 19:11:09.892899 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.892888    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64263691-60bb-4559-86ad-0ec935f6357a-maas-api-tls\") pod \"maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh\" (UID: \"64263691-60bb-4559-86ad-0ec935f6357a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"
Sep 15 19:11:09.898505 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.898478    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbvwh\" (UniqueName: \"kubernetes.io/projected/64263691-60bb-4559-86ad-0ec935f6357a-kube-api-access-vbvwh\") pod \"maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh\" (UID: \"64263691-60bb-4559-86ad-0ec935f6357a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"
Sep 15 19:11:09.973745 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:09.973639    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"
Sep 15 19:11:10.255518 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:10.255482    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"]
Sep 15 19:11:10.259116 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:11:10.259084    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64263691_60bb_4559_86ad_0ec935f6357a.slice/crio-a83e2d04cd2ea107491f1ab411c9dda8f1c77374dda11f92d38f1487227d142e WatchSource:0}: Error finding container a83e2d04cd2ea107491f1ab411c9dda8f1c77374dda11f92d38f1487227d142e: Status 404 returned error can't find the container with id a83e2d04cd2ea107491f1ab411c9dda8f1c77374dda11f92d38f1487227d142e
Sep 15 19:11:11.226982 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:11.226949    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh" event={"ID":"64263691-60bb-4559-86ad-0ec935f6357a","Type":"ContainerStarted","Data":"9e0397c61b080ed933b65d86794ad5d37afae08d1bba233b23406cc1a46aa9e0"}
Sep 15 19:11:11.227348 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:11.226995    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh" event={"ID":"64263691-60bb-4559-86ad-0ec935f6357a","Type":"ContainerStarted","Data":"a83e2d04cd2ea107491f1ab411c9dda8f1c77374dda11f92d38f1487227d142e"}
Sep 15 19:11:11.227348 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:11.227029    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"
Sep 15 19:11:11.243239 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:11.243193    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh" podStartSLOduration=1.861671328 podStartE2EDuration="2.24317831s" totalImagesPullingTime="381.506982ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:11:11.241441836 +0000 UTC m=+1212.728770822" watchObservedRunningTime="2026-09-15 19:11:11.24317831 +0000 UTC m=+1212.730507296"
Sep 15 19:11:17.238674 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:17.238639    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"
Sep 15 19:11:18.416706 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:18.416671    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"]
Sep 15 19:11:18.417171 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:18.416932    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb" podUID="c5efbd9d-fd2d-473d-921d-7adad6724f90" containerName="maas-api" containerID="cri-o://079fba945a6dc6ba204fe439585aabb7e156fb16475b9a4cf40ffc37227b1b67" gracePeriod=30
Sep 15 19:11:18.868556 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:18.868528    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"
Sep 15 19:11:18.894651 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:18.894099    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5efbd9d-fd2d-473d-921d-7adad6724f90-metrics-certs\") pod \"c5efbd9d-fd2d-473d-921d-7adad6724f90\" (UID: \"c5efbd9d-fd2d-473d-921d-7adad6724f90\") "
Sep 15 19:11:18.894651 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:18.894217    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5efbd9d-fd2d-473d-921d-7adad6724f90-openshift-service-ca\") pod \"c5efbd9d-fd2d-473d-921d-7adad6724f90\" (UID: \"c5efbd9d-fd2d-473d-921d-7adad6724f90\") "
Sep 15 19:11:18.894651 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:18.894246    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5efbd9d-fd2d-473d-921d-7adad6724f90-maas-api-tls\") pod \"c5efbd9d-fd2d-473d-921d-7adad6724f90\" (UID: \"c5efbd9d-fd2d-473d-921d-7adad6724f90\") "
Sep 15 19:11:18.894651 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:18.894336    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kth88\" (UniqueName: \"kubernetes.io/projected/c5efbd9d-fd2d-473d-921d-7adad6724f90-kube-api-access-kth88\") pod \"c5efbd9d-fd2d-473d-921d-7adad6724f90\" (UID: \"c5efbd9d-fd2d-473d-921d-7adad6724f90\") "
Sep 15 19:11:18.895785 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:18.895760    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5efbd9d-fd2d-473d-921d-7adad6724f90-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c5efbd9d-fd2d-473d-921d-7adad6724f90" (UID: "c5efbd9d-fd2d-473d-921d-7adad6724f90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:11:18.900793 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:18.900751    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5efbd9d-fd2d-473d-921d-7adad6724f90-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c5efbd9d-fd2d-473d-921d-7adad6724f90" (UID: "c5efbd9d-fd2d-473d-921d-7adad6724f90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:11:18.908859 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:18.908758    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5efbd9d-fd2d-473d-921d-7adad6724f90-kube-api-access-kth88" (OuterVolumeSpecName: "kube-api-access-kth88") pod "c5efbd9d-fd2d-473d-921d-7adad6724f90" (UID: "c5efbd9d-fd2d-473d-921d-7adad6724f90"). InnerVolumeSpecName "kube-api-access-kth88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:11:18.908859 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:18.908829    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5efbd9d-fd2d-473d-921d-7adad6724f90-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c5efbd9d-fd2d-473d-921d-7adad6724f90" (UID: "c5efbd9d-fd2d-473d-921d-7adad6724f90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:11:18.996650 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:18.996591    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5efbd9d-fd2d-473d-921d-7adad6724f90-openshift-service-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:18.996650 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:18.996646    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5efbd9d-fd2d-473d-921d-7adad6724f90-maas-api-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:18.996650 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:18.996655    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kth88\" (UniqueName: \"kubernetes.io/projected/c5efbd9d-fd2d-473d-921d-7adad6724f90-kube-api-access-kth88\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:18.996870 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:18.996663    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5efbd9d-fd2d-473d-921d-7adad6724f90-metrics-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:19.274754 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:19.274665    2555 generic.go:356] "Generic (PLEG): container finished" podID="c5efbd9d-fd2d-473d-921d-7adad6724f90" containerID="079fba945a6dc6ba204fe439585aabb7e156fb16475b9a4cf40ffc37227b1b67" exitCode=0
Sep 15 19:11:19.274754 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:19.274714    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"
Sep 15 19:11:19.274974 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:19.274719    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb" event={"ID":"c5efbd9d-fd2d-473d-921d-7adad6724f90","Type":"ContainerDied","Data":"079fba945a6dc6ba204fe439585aabb7e156fb16475b9a4cf40ffc37227b1b67"}
Sep 15 19:11:19.274974 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:19.274827    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb" event={"ID":"c5efbd9d-fd2d-473d-921d-7adad6724f90","Type":"ContainerDied","Data":"88188defa8aa6ffbed0f6340ef380bc9def6d14d02d33d527d0e74cb4061bb04"}
Sep 15 19:11:19.274974 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:19.274850    2555 scope.go:117] "RemoveContainer" containerID="079fba945a6dc6ba204fe439585aabb7e156fb16475b9a4cf40ffc37227b1b67"
Sep 15 19:11:19.284050 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:19.284028    2555 scope.go:117] "RemoveContainer" containerID="079fba945a6dc6ba204fe439585aabb7e156fb16475b9a4cf40ffc37227b1b67"
Sep 15 19:11:19.284348 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:11:19.284325    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"079fba945a6dc6ba204fe439585aabb7e156fb16475b9a4cf40ffc37227b1b67\": container with ID starting with 079fba945a6dc6ba204fe439585aabb7e156fb16475b9a4cf40ffc37227b1b67 not found: ID does not exist" containerID="079fba945a6dc6ba204fe439585aabb7e156fb16475b9a4cf40ffc37227b1b67"
Sep 15 19:11:19.284415 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:19.284354    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"079fba945a6dc6ba204fe439585aabb7e156fb16475b9a4cf40ffc37227b1b67"} err="failed to get container status \"079fba945a6dc6ba204fe439585aabb7e156fb16475b9a4cf40ffc37227b1b67\": rpc error: code = NotFound desc = could not find container \"079fba945a6dc6ba204fe439585aabb7e156fb16475b9a4cf40ffc37227b1b67\": container with ID starting with 079fba945a6dc6ba204fe439585aabb7e156fb16475b9a4cf40ffc37227b1b67 not found: ID does not exist"
Sep 15 19:11:19.290508 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:19.290483    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"]
Sep 15 19:11:19.293769 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:19.293749    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7c96ca-76bf69567b-4qfdb"]
Sep 15 19:11:21.041240 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:21.041206    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5efbd9d-fd2d-473d-921d-7adad6724f90" path="/var/lib/kubelet/pods/c5efbd9d-fd2d-473d-921d-7adad6724f90/volumes"
Sep 15 19:11:22.608931 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.608881    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d" podUID="f81db17b-7991-4f3f-a51d-45158ff78d14" containerName="main" containerID="cri-o://41e49e80b5629b7367ff83ce446f10917567bc87c9a2e19f8d35231a2c5394e8" gracePeriod=45
Sep 15 19:11:22.622666 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.622633    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq" podUID="0169d288-9a3a-4738-a6e6-922210087864" containerName="main" containerID="cri-o://ae14218c1f1a316754a0b193db3b3eb97e708b07ead8fca9cf2f3cb25401244d" gracePeriod=45
Sep 15 19:11:22.633329 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.633297    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c" podUID="2cb4e1f6-31d0-4fab-9f08-1dd91c819725" containerName="main" containerID="cri-o://82641bde1e236c15f7e1b68e54337621d85b2be35af5dbd24eff72471146d224" gracePeriod=45
Sep 15 19:11:22.891987 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.891961    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:11:22.932113 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.932086    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:11:22.935862 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.935841    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:11:22.936049 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.936032    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-tmp-dir\") pod \"0169d288-9a3a-4738-a6e6-922210087864\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") "
Sep 15 19:11:22.936133 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.936059    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-home\") pod \"0169d288-9a3a-4738-a6e6-922210087864\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") "
Sep 15 19:11:22.936133 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.936081    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-dshm\") pod \"0169d288-9a3a-4738-a6e6-922210087864\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") "
Sep 15 19:11:22.936133 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.936117    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-model-cache\") pod \"0169d288-9a3a-4738-a6e6-922210087864\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") "
Sep 15 19:11:22.936236 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.936180    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw86w\" (UniqueName: \"kubernetes.io/projected/0169d288-9a3a-4738-a6e6-922210087864-kube-api-access-mw86w\") pod \"0169d288-9a3a-4738-a6e6-922210087864\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") "
Sep 15 19:11:22.936236 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.936228    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0169d288-9a3a-4738-a6e6-922210087864-tls-certs\") pod \"0169d288-9a3a-4738-a6e6-922210087864\" (UID: \"0169d288-9a3a-4738-a6e6-922210087864\") "
Sep 15 19:11:22.936664 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.936342    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-home" (OuterVolumeSpecName: "home") pod "0169d288-9a3a-4738-a6e6-922210087864" (UID: "0169d288-9a3a-4738-a6e6-922210087864"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:11:22.936664 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.936477    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-model-cache" (OuterVolumeSpecName: "model-cache") pod "0169d288-9a3a-4738-a6e6-922210087864" (UID: "0169d288-9a3a-4738-a6e6-922210087864"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:11:22.936664 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.936528    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0169d288-9a3a-4738-a6e6-922210087864" (UID: "0169d288-9a3a-4738-a6e6-922210087864"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:11:22.936897 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.936688    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-tmp-dir\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:22.936897 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.936705    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-home\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:22.936897 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.936718    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-model-cache\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:22.938681 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.938657    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0169d288-9a3a-4738-a6e6-922210087864-kube-api-access-mw86w" (OuterVolumeSpecName: "kube-api-access-mw86w") pod "0169d288-9a3a-4738-a6e6-922210087864" (UID: "0169d288-9a3a-4738-a6e6-922210087864"). InnerVolumeSpecName "kube-api-access-mw86w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:11:22.938761 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.938738    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-dshm" (OuterVolumeSpecName: "dshm") pod "0169d288-9a3a-4738-a6e6-922210087864" (UID: "0169d288-9a3a-4738-a6e6-922210087864"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:11:22.938864 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:22.938847    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0169d288-9a3a-4738-a6e6-922210087864-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0169d288-9a3a-4738-a6e6-922210087864" (UID: "0169d288-9a3a-4738-a6e6-922210087864"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:11:23.037030 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037001    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-tmp-dir\") pod \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") "
Sep 15 19:11:23.037226 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037076    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmz4g\" (UniqueName: \"kubernetes.io/projected/f81db17b-7991-4f3f-a51d-45158ff78d14-kube-api-access-bmz4g\") pod \"f81db17b-7991-4f3f-a51d-45158ff78d14\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") "
Sep 15 19:11:23.037226 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037117    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-home\") pod \"f81db17b-7991-4f3f-a51d-45158ff78d14\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") "
Sep 15 19:11:23.037226 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037145    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-tmp-dir\") pod \"f81db17b-7991-4f3f-a51d-45158ff78d14\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") "
Sep 15 19:11:23.037226 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037181    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw4ng\" (UniqueName: \"kubernetes.io/projected/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-kube-api-access-nw4ng\") pod \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") "
Sep 15 19:11:23.037226 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037211    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-model-cache\") pod \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") "
Sep 15 19:11:23.037517 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037267    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-dshm\") pod \"f81db17b-7991-4f3f-a51d-45158ff78d14\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") "
Sep 15 19:11:23.037517 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037297    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-model-cache\") pod \"f81db17b-7991-4f3f-a51d-45158ff78d14\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") "
Sep 15 19:11:23.037517 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037324    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-dshm\") pod \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") "
Sep 15 19:11:23.037517 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037355    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f81db17b-7991-4f3f-a51d-45158ff78d14-tls-certs\") pod \"f81db17b-7991-4f3f-a51d-45158ff78d14\" (UID: \"f81db17b-7991-4f3f-a51d-45158ff78d14\") "
Sep 15 19:11:23.037517 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037402    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-home\") pod \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") "
Sep 15 19:11:23.037517 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037473    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-tls-certs\") pod \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\" (UID: \"2cb4e1f6-31d0-4fab-9f08-1dd91c819725\") "
Sep 15 19:11:23.037517 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037292    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2cb4e1f6-31d0-4fab-9f08-1dd91c819725" (UID: "2cb4e1f6-31d0-4fab-9f08-1dd91c819725"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:11:23.037517 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037478    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-model-cache" (OuterVolumeSpecName: "model-cache") pod "2cb4e1f6-31d0-4fab-9f08-1dd91c819725" (UID: "2cb4e1f6-31d0-4fab-9f08-1dd91c819725"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:11:23.037517 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037498    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f81db17b-7991-4f3f-a51d-45158ff78d14" (UID: "f81db17b-7991-4f3f-a51d-45158ff78d14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:11:23.037992 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037847    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0169d288-9a3a-4738-a6e6-922210087864-dshm\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.037992 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037864    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-tmp-dir\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.037992 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037877    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw86w\" (UniqueName: \"kubernetes.io/projected/0169d288-9a3a-4738-a6e6-922210087864-kube-api-access-mw86w\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.037992 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037892    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0169d288-9a3a-4738-a6e6-922210087864-tls-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.037992 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037905    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-tmp-dir\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.037992 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037916    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-model-cache\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.037992 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.037915    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-home" (OuterVolumeSpecName: "home") pod "2cb4e1f6-31d0-4fab-9f08-1dd91c819725" (UID: "2cb4e1f6-31d0-4fab-9f08-1dd91c819725"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:11:23.039677 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.039652    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-home" (OuterVolumeSpecName: "home") pod "f81db17b-7991-4f3f-a51d-45158ff78d14" (UID: "f81db17b-7991-4f3f-a51d-45158ff78d14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:11:23.039901 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.039875    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-model-cache" (OuterVolumeSpecName: "model-cache") pod "f81db17b-7991-4f3f-a51d-45158ff78d14" (UID: "f81db17b-7991-4f3f-a51d-45158ff78d14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:11:23.039901 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.039879    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f81db17b-7991-4f3f-a51d-45158ff78d14-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f81db17b-7991-4f3f-a51d-45158ff78d14" (UID: "f81db17b-7991-4f3f-a51d-45158ff78d14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:11:23.040063 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.039990    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-dshm" (OuterVolumeSpecName: "dshm") pod "f81db17b-7991-4f3f-a51d-45158ff78d14" (UID: "f81db17b-7991-4f3f-a51d-45158ff78d14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:11:23.040304 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.040274    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2cb4e1f6-31d0-4fab-9f08-1dd91c819725" (UID: "2cb4e1f6-31d0-4fab-9f08-1dd91c819725"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:11:23.040499 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.040469    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81db17b-7991-4f3f-a51d-45158ff78d14-kube-api-access-bmz4g" (OuterVolumeSpecName: "kube-api-access-bmz4g") pod "f81db17b-7991-4f3f-a51d-45158ff78d14" (UID: "f81db17b-7991-4f3f-a51d-45158ff78d14"). InnerVolumeSpecName "kube-api-access-bmz4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:11:23.041817 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.041785    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-kube-api-access-nw4ng" (OuterVolumeSpecName: "kube-api-access-nw4ng") pod "2cb4e1f6-31d0-4fab-9f08-1dd91c819725" (UID: "2cb4e1f6-31d0-4fab-9f08-1dd91c819725"). InnerVolumeSpecName "kube-api-access-nw4ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:11:23.041940 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.041924    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-dshm" (OuterVolumeSpecName: "dshm") pod "2cb4e1f6-31d0-4fab-9f08-1dd91c819725" (UID: "2cb4e1f6-31d0-4fab-9f08-1dd91c819725"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:11:23.070617 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:11:23.070582    2555 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf81db17b_7991_4f3f_a51d_45158ff78d14.slice\": RecentStats: unable to find data in memory cache]"
Sep 15 19:11:23.138981 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.138899    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-tls-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.138981 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.138928    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmz4g\" (UniqueName: \"kubernetes.io/projected/f81db17b-7991-4f3f-a51d-45158ff78d14-kube-api-access-bmz4g\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.138981 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.138937    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-home\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.138981 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.138947    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw4ng\" (UniqueName: \"kubernetes.io/projected/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-kube-api-access-nw4ng\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.138981 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.138955    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-dshm\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.138981 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.138963    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f81db17b-7991-4f3f-a51d-45158ff78d14-model-cache\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.138981 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.138971    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-dshm\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.138981 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.138978    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f81db17b-7991-4f3f-a51d-45158ff78d14-tls-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.138981 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.138985    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb4e1f6-31d0-4fab-9f08-1dd91c819725-home\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:11:23.300111 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.300076    2555 generic.go:356] "Generic (PLEG): container finished" podID="2cb4e1f6-31d0-4fab-9f08-1dd91c819725" containerID="82641bde1e236c15f7e1b68e54337621d85b2be35af5dbd24eff72471146d224" exitCode=0
Sep 15 19:11:23.300307 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.300139    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"
Sep 15 19:11:23.300307 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.300163    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c" event={"ID":"2cb4e1f6-31d0-4fab-9f08-1dd91c819725","Type":"ContainerDied","Data":"82641bde1e236c15f7e1b68e54337621d85b2be35af5dbd24eff72471146d224"}
Sep 15 19:11:23.300307 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.300201    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c" event={"ID":"2cb4e1f6-31d0-4fab-9f08-1dd91c819725","Type":"ContainerDied","Data":"5ae03f48bb1dd62c78db885a19b0101cd1e167d4a7dda0841dd15c3e983a659f"}
Sep 15 19:11:23.300307 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.300218    2555 scope.go:117] "RemoveContainer" containerID="82641bde1e236c15f7e1b68e54337621d85b2be35af5dbd24eff72471146d224"
Sep 15 19:11:23.301791 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.301767    2555 generic.go:356] "Generic (PLEG): container finished" podID="0169d288-9a3a-4738-a6e6-922210087864" containerID="ae14218c1f1a316754a0b193db3b3eb97e708b07ead8fca9cf2f3cb25401244d" exitCode=0
Sep 15 19:11:23.301891 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.301816    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"
Sep 15 19:11:23.301891 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.301847    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq" event={"ID":"0169d288-9a3a-4738-a6e6-922210087864","Type":"ContainerDied","Data":"ae14218c1f1a316754a0b193db3b3eb97e708b07ead8fca9cf2f3cb25401244d"}
Sep 15 19:11:23.301891 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.301883    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq" event={"ID":"0169d288-9a3a-4738-a6e6-922210087864","Type":"ContainerDied","Data":"65bcf8e661f660d573e9c5e06d19431b76158c29adccf6b5303dcdb40adfa82c"}
Sep 15 19:11:23.303371 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.303349    2555 generic.go:356] "Generic (PLEG): container finished" podID="f81db17b-7991-4f3f-a51d-45158ff78d14" containerID="41e49e80b5629b7367ff83ce446f10917567bc87c9a2e19f8d35231a2c5394e8" exitCode=0
Sep 15 19:11:23.303483 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.303406    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"
Sep 15 19:11:23.303483 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.303440    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d" event={"ID":"f81db17b-7991-4f3f-a51d-45158ff78d14","Type":"ContainerDied","Data":"41e49e80b5629b7367ff83ce446f10917567bc87c9a2e19f8d35231a2c5394e8"}
Sep 15 19:11:23.303483 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.303470    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d" event={"ID":"f81db17b-7991-4f3f-a51d-45158ff78d14","Type":"ContainerDied","Data":"0e8b512d69e09b49989a39208edfd5a04ebbb14e564e4d78008af09c3c2c69ea"}
Sep 15 19:11:23.310453 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.310439    2555 scope.go:117] "RemoveContainer" containerID="82641bde1e236c15f7e1b68e54337621d85b2be35af5dbd24eff72471146d224"
Sep 15 19:11:23.310841 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:11:23.310820    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82641bde1e236c15f7e1b68e54337621d85b2be35af5dbd24eff72471146d224\": container with ID starting with 82641bde1e236c15f7e1b68e54337621d85b2be35af5dbd24eff72471146d224 not found: ID does not exist" containerID="82641bde1e236c15f7e1b68e54337621d85b2be35af5dbd24eff72471146d224"
Sep 15 19:11:23.310913 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.310846    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82641bde1e236c15f7e1b68e54337621d85b2be35af5dbd24eff72471146d224"} err="failed to get container status \"82641bde1e236c15f7e1b68e54337621d85b2be35af5dbd24eff72471146d224\": rpc error: code = NotFound desc = could not find container \"82641bde1e236c15f7e1b68e54337621d85b2be35af5dbd24eff72471146d224\": container with ID starting with 82641bde1e236c15f7e1b68e54337621d85b2be35af5dbd24eff72471146d224 not found: ID does not exist"
Sep 15 19:11:23.310913 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.310863    2555 scope.go:117] "RemoveContainer" containerID="ae14218c1f1a316754a0b193db3b3eb97e708b07ead8fca9cf2f3cb25401244d"
Sep 15 19:11:23.318816 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.318799    2555 scope.go:117] "RemoveContainer" containerID="ae14218c1f1a316754a0b193db3b3eb97e708b07ead8fca9cf2f3cb25401244d"
Sep 15 19:11:23.319070 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:11:23.319054    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae14218c1f1a316754a0b193db3b3eb97e708b07ead8fca9cf2f3cb25401244d\": container with ID starting with ae14218c1f1a316754a0b193db3b3eb97e708b07ead8fca9cf2f3cb25401244d not found: ID does not exist" containerID="ae14218c1f1a316754a0b193db3b3eb97e708b07ead8fca9cf2f3cb25401244d"
Sep 15 19:11:23.319111 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.319076    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae14218c1f1a316754a0b193db3b3eb97e708b07ead8fca9cf2f3cb25401244d"} err="failed to get container status \"ae14218c1f1a316754a0b193db3b3eb97e708b07ead8fca9cf2f3cb25401244d\": rpc error: code = NotFound desc = could not find container \"ae14218c1f1a316754a0b193db3b3eb97e708b07ead8fca9cf2f3cb25401244d\": container with ID starting with ae14218c1f1a316754a0b193db3b3eb97e708b07ead8fca9cf2f3cb25401244d not found: ID does not exist"
Sep 15 19:11:23.319111 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.319089    2555 scope.go:117] "RemoveContainer" containerID="41e49e80b5629b7367ff83ce446f10917567bc87c9a2e19f8d35231a2c5394e8"
Sep 15 19:11:23.326482 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.326462    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"]
Sep 15 19:11:23.328320 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.328305    2555 scope.go:117] "RemoveContainer" containerID="41e49e80b5629b7367ff83ce446f10917567bc87c9a2e19f8d35231a2c5394e8"
Sep 15 19:11:23.328634 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:11:23.328595    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e49e80b5629b7367ff83ce446f10917567bc87c9a2e19f8d35231a2c5394e8\": container with ID starting with 41e49e80b5629b7367ff83ce446f10917567bc87c9a2e19f8d35231a2c5394e8 not found: ID does not exist" containerID="41e49e80b5629b7367ff83ce446f10917567bc87c9a2e19f8d35231a2c5394e8"
Sep 15 19:11:23.328704 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.328641    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e49e80b5629b7367ff83ce446f10917567bc87c9a2e19f8d35231a2c5394e8"} err="failed to get container status \"41e49e80b5629b7367ff83ce446f10917567bc87c9a2e19f8d35231a2c5394e8\": rpc error: code = NotFound desc = could not find container \"41e49e80b5629b7367ff83ce446f10917567bc87c9a2e19f8d35231a2c5394e8\": container with ID starting with 41e49e80b5629b7367ff83ce446f10917567bc87c9a2e19f8d35231a2c5394e8 not found: ID does not exist"
Sep 15 19:11:23.329232 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.329215    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/distinct-w3-7c96ca-kserve-6c45675b77-kr79d"]
Sep 15 19:11:23.337581 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.337526    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"]
Sep 15 19:11:23.339395 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.339377    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/premium-w3-7c96ca-kserve-844d8d98-kpqpq"]
Sep 15 19:11:23.348975 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.348954    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"]
Sep 15 19:11:23.354003 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:23.353978    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7c96ca/simulator-w3-7c96ca-kserve-66dc48bb47-stl4c"]
Sep 15 19:11:25.041787 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:25.041759    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0169d288-9a3a-4738-a6e6-922210087864" path="/var/lib/kubelet/pods/0169d288-9a3a-4738-a6e6-922210087864/volumes"
Sep 15 19:11:25.042219 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:25.042201    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cb4e1f6-31d0-4fab-9f08-1dd91c819725" path="/var/lib/kubelet/pods/2cb4e1f6-31d0-4fab-9f08-1dd91c819725/volumes"
Sep 15 19:11:25.042596 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:25.042584    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f81db17b-7991-4f3f-a51d-45158ff78d14" path="/var/lib/kubelet/pods/f81db17b-7991-4f3f-a51d-45158ff78d14/volumes"
Sep 15 19:11:52.535005 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.534933    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"]
Sep 15 19:11:52.535353 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.535341    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f81db17b-7991-4f3f-a51d-45158ff78d14" containerName="main"
Sep 15 19:11:52.535353 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.535354    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f81db17b-7991-4f3f-a51d-45158ff78d14" containerName="main"
Sep 15 19:11:52.535430 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.535363    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cb4e1f6-31d0-4fab-9f08-1dd91c819725" containerName="main"
Sep 15 19:11:52.535430 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.535369    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cb4e1f6-31d0-4fab-9f08-1dd91c819725" containerName="main"
Sep 15 19:11:52.535430 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.535379    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5efbd9d-fd2d-473d-921d-7adad6724f90" containerName="maas-api"
Sep 15 19:11:52.535430 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.535384    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5efbd9d-fd2d-473d-921d-7adad6724f90" containerName="maas-api"
Sep 15 19:11:52.535430 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.535407    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0169d288-9a3a-4738-a6e6-922210087864" containerName="main"
Sep 15 19:11:52.535430 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.535412    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0169d288-9a3a-4738-a6e6-922210087864" containerName="main"
Sep 15 19:11:52.535634 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.535472    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cb4e1f6-31d0-4fab-9f08-1dd91c819725" containerName="main"
Sep 15 19:11:52.535634 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.535479    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5efbd9d-fd2d-473d-921d-7adad6724f90" containerName="maas-api"
Sep 15 19:11:52.535634 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.535487    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0169d288-9a3a-4738-a6e6-922210087864" containerName="main"
Sep 15 19:11:52.535634 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.535496    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f81db17b-7991-4f3f-a51d-45158ff78d14" containerName="main"
Sep 15 19:11:52.538918 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.538899    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:52.542677 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.542654    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-33ff04\"/\"default-dockercfg-9prxg\""
Sep 15 19:11:52.542784 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.542681    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-33ff04\"/\"openshift-service-ca.crt\""
Sep 15 19:11:52.542784 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.542657    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-33ff04\"/\"kube-root-ca.crt\""
Sep 15 19:11:52.542784 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.542657    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-33ff04\"/\"auth-test-model-33ff04-kserve-self-signed-certs\""
Sep 15 19:11:52.555098 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.555076    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"]
Sep 15 19:11:52.641471 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.641436    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-tmp-dir\") pod \"auth-test-model-33ff04-kserve-56496684b7-tbs2c\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:52.641717 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.641543    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvpxc\" (UniqueName: \"kubernetes.io/projected/33731e0d-0c55-4977-b8a0-5f8befa9a27a-kube-api-access-vvpxc\") pod \"auth-test-model-33ff04-kserve-56496684b7-tbs2c\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:52.641717 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.641582    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33731e0d-0c55-4977-b8a0-5f8befa9a27a-tls-certs\") pod \"auth-test-model-33ff04-kserve-56496684b7-tbs2c\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:52.641717 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.641636    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-home\") pod \"auth-test-model-33ff04-kserve-56496684b7-tbs2c\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:52.641717 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.641669    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-model-cache\") pod \"auth-test-model-33ff04-kserve-56496684b7-tbs2c\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:52.641943 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.641784    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-dshm\") pod \"auth-test-model-33ff04-kserve-56496684b7-tbs2c\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:52.742335 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.742297    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-dshm\") pod \"auth-test-model-33ff04-kserve-56496684b7-tbs2c\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:52.742530 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.742348    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-tmp-dir\") pod \"auth-test-model-33ff04-kserve-56496684b7-tbs2c\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:52.742530 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.742403    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvpxc\" (UniqueName: \"kubernetes.io/projected/33731e0d-0c55-4977-b8a0-5f8befa9a27a-kube-api-access-vvpxc\") pod \"auth-test-model-33ff04-kserve-56496684b7-tbs2c\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:52.742530 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.742429    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33731e0d-0c55-4977-b8a0-5f8befa9a27a-tls-certs\") pod \"auth-test-model-33ff04-kserve-56496684b7-tbs2c\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:52.742530 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.742445    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-home\") pod \"auth-test-model-33ff04-kserve-56496684b7-tbs2c\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:52.742530 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.742474    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-model-cache\") pod \"auth-test-model-33ff04-kserve-56496684b7-tbs2c\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:52.742850 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.742826    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-tmp-dir\") pod \"auth-test-model-33ff04-kserve-56496684b7-tbs2c\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:52.742918 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.742860    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-model-cache\") pod \"auth-test-model-33ff04-kserve-56496684b7-tbs2c\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:52.742918 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.742899    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-home\") pod \"auth-test-model-33ff04-kserve-56496684b7-tbs2c\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:52.744534 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.744512    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-dshm\") pod \"auth-test-model-33ff04-kserve-56496684b7-tbs2c\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:52.744943 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.744928    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33731e0d-0c55-4977-b8a0-5f8befa9a27a-tls-certs\") pod \"auth-test-model-33ff04-kserve-56496684b7-tbs2c\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:52.749404 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.749383    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvpxc\" (UniqueName: \"kubernetes.io/projected/33731e0d-0c55-4977-b8a0-5f8befa9a27a-kube-api-access-vvpxc\") pod \"auth-test-model-33ff04-kserve-56496684b7-tbs2c\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:52.847978 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:52.847890    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:11:53.121203 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:53.121162    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"]
Sep 15 19:11:53.122651 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:11:53.122626    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33731e0d_0c55_4977_b8a0_5f8befa9a27a.slice/crio-19a9ed0895c80f54e4aa0e520f05be3a732c0ec0b8951bbd4ecca3db53052242 WatchSource:0}: Error finding container 19a9ed0895c80f54e4aa0e520f05be3a732c0ec0b8951bbd4ecca3db53052242: Status 404 returned error can't find the container with id 19a9ed0895c80f54e4aa0e520f05be3a732c0ec0b8951bbd4ecca3db53052242
Sep 15 19:11:53.473683 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:53.473579    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c" event={"ID":"33731e0d-0c55-4977-b8a0-5f8befa9a27a","Type":"ContainerStarted","Data":"14ad6e5521e98479373e4a54ffc14b87169561436439dd0dd366dd5931fe9a9b"}
Sep 15 19:11:53.473683 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:53.473638    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c" event={"ID":"33731e0d-0c55-4977-b8a0-5f8befa9a27a","Type":"ContainerStarted","Data":"19a9ed0895c80f54e4aa0e520f05be3a732c0ec0b8951bbd4ecca3db53052242"}
Sep 15 19:11:53.490698 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:53.490644    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c" podStartSLOduration=1.490628926 podStartE2EDuration="1.490628926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:11:53.488403316 +0000 UTC m=+1254.975732298" watchObservedRunningTime="2026-09-15 19:11:53.490628926 +0000 UTC m=+1254.977957908"
Sep 15 19:11:59.120989 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:11:59.120954    2555 scope.go:117] "RemoveContainer" containerID="bbc66431da5d4a29f3023700c475f1728a69d01ed2d30b7c062f2a9e5dd70886"
Sep 15 19:12:02.848023 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:02.847985    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:12:02.848023 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:02.848028    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:12:02.860734 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:02.860698    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:12:03.549412 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:03.549382    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:12:05.840767 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:05.840728    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"]
Sep 15 19:12:05.846164 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:05.846136    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:05.849314 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:05.849291    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d2e6cc70\"/\"kube-root-ca.crt\""
Sep 15 19:12:05.849464 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:05.849317    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d2e6cc70\"/\"e2e-lifecycle-model-d2e6cc70-kserve-self-signed-certs\""
Sep 15 19:12:05.849464 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:05.849319    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d2e6cc70\"/\"openshift-service-ca.crt\""
Sep 15 19:12:05.849464 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:05.849372    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d2e6cc70\"/\"default-dockercfg-b4v9x\""
Sep 15 19:12:05.853564 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:05.853541    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"]
Sep 15 19:12:05.981786 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:05.981749    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-dshm\") pod \"e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") " pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:05.981992 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:05.981810    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxzsv\" (UniqueName: \"kubernetes.io/projected/03e11035-89d4-42ed-9a1b-13d343f3ce7a-kube-api-access-mxzsv\") pod \"e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") " pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:05.981992 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:05.981892    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-tmp-dir\") pod \"e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") " pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:05.981992 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:05.981951    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-home\") pod \"e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") " pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:05.982108 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:05.982019    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-model-cache\") pod \"e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") " pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:05.982108 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:05.982080    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e11035-89d4-42ed-9a1b-13d343f3ce7a-tls-certs\") pod \"e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") " pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:06.083310 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:06.083275    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e11035-89d4-42ed-9a1b-13d343f3ce7a-tls-certs\") pod \"e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") " pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:06.083514 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:06.083322    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-dshm\") pod \"e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") " pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:06.083514 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:06.083350    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxzsv\" (UniqueName: \"kubernetes.io/projected/03e11035-89d4-42ed-9a1b-13d343f3ce7a-kube-api-access-mxzsv\") pod \"e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") " pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:06.083514 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:06.083412    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-tmp-dir\") pod \"e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") " pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:06.083514 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:06.083469    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-home\") pod \"e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") " pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:06.083514 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:06.083504    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-model-cache\") pod \"e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") " pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:06.083878 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:06.083852    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-tmp-dir\") pod \"e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") " pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:06.083950 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:06.083863    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-home\") pod \"e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") " pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:06.083950 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:06.083919    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-model-cache\") pod \"e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") " pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:06.085671 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:06.085652    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-dshm\") pod \"e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") " pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:06.086047 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:06.086026    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e11035-89d4-42ed-9a1b-13d343f3ce7a-tls-certs\") pod \"e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") " pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:06.092979 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:06.092914    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxzsv\" (UniqueName: \"kubernetes.io/projected/03e11035-89d4-42ed-9a1b-13d343f3ce7a-kube-api-access-mxzsv\") pod \"e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") " pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:06.156894 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:06.156857    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:06.432048 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:06.429693    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"]
Sep 15 19:12:06.432721 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:12:06.432695    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03e11035_89d4_42ed_9a1b_13d343f3ce7a.slice/crio-2d6145075be6ced5c570d41f1a981e5be6a16af5b419b13ecba5da00dde0aab2 WatchSource:0}: Error finding container 2d6145075be6ced5c570d41f1a981e5be6a16af5b419b13ecba5da00dde0aab2: Status 404 returned error can't find the container with id 2d6145075be6ced5c570d41f1a981e5be6a16af5b419b13ecba5da00dde0aab2
Sep 15 19:12:06.555760 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:06.555725    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8" event={"ID":"03e11035-89d4-42ed-9a1b-13d343f3ce7a","Type":"ContainerStarted","Data":"02e2f4a0fde4b26553686dafdc0e66148384fbafb390e649e36d777c3ebcd5a7"}
Sep 15 19:12:06.555760 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:06.555765    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8" event={"ID":"03e11035-89d4-42ed-9a1b-13d343f3ce7a","Type":"ContainerStarted","Data":"2d6145075be6ced5c570d41f1a981e5be6a16af5b419b13ecba5da00dde0aab2"}
Sep 15 19:12:06.572833 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:06.572772    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8" podStartSLOduration=1.5727504319999999 podStartE2EDuration="1.572750432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:12:06.571206526 +0000 UTC m=+1268.058535512" watchObservedRunningTime="2026-09-15 19:12:06.572750432 +0000 UTC m=+1268.060079418"
Sep 15 19:12:16.157675 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:16.157632    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:16.158190 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:16.157696    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:16.170484 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:16.170457    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:12:16.625011 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:12:16.624982    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:13:23.650586 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:23.650546    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"]
Sep 15 19:13:23.651095 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:23.650888    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh" podUID="64263691-60bb-4559-86ad-0ec935f6357a" containerName="maas-api" containerID="cri-o://9e0397c61b080ed933b65d86794ad5d37afae08d1bba233b23406cc1a46aa9e0" gracePeriod=30
Sep 15 19:13:23.905304 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:23.905235    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"
Sep 15 19:13:23.984290 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:23.984258    2555 generic.go:356] "Generic (PLEG): container finished" podID="64263691-60bb-4559-86ad-0ec935f6357a" containerID="9e0397c61b080ed933b65d86794ad5d37afae08d1bba233b23406cc1a46aa9e0" exitCode=0
Sep 15 19:13:23.984480 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:23.984310    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"
Sep 15 19:13:23.984480 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:23.984311    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh" event={"ID":"64263691-60bb-4559-86ad-0ec935f6357a","Type":"ContainerDied","Data":"9e0397c61b080ed933b65d86794ad5d37afae08d1bba233b23406cc1a46aa9e0"}
Sep 15 19:13:23.984480 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:23.984474    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh" event={"ID":"64263691-60bb-4559-86ad-0ec935f6357a","Type":"ContainerDied","Data":"a83e2d04cd2ea107491f1ab411c9dda8f1c77374dda11f92d38f1487227d142e"}
Sep 15 19:13:23.984619 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:23.984506    2555 scope.go:117] "RemoveContainer" containerID="9e0397c61b080ed933b65d86794ad5d37afae08d1bba233b23406cc1a46aa9e0"
Sep 15 19:13:23.986882 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:23.986864    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbvwh\" (UniqueName: \"kubernetes.io/projected/64263691-60bb-4559-86ad-0ec935f6357a-kube-api-access-vbvwh\") pod \"64263691-60bb-4559-86ad-0ec935f6357a\" (UID: \"64263691-60bb-4559-86ad-0ec935f6357a\") "
Sep 15 19:13:23.986982 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:23.986921    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64263691-60bb-4559-86ad-0ec935f6357a-openshift-service-ca\") pod \"64263691-60bb-4559-86ad-0ec935f6357a\" (UID: \"64263691-60bb-4559-86ad-0ec935f6357a\") "
Sep 15 19:13:23.987051 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:23.987020    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64263691-60bb-4559-86ad-0ec935f6357a-metrics-certs\") pod \"64263691-60bb-4559-86ad-0ec935f6357a\" (UID: \"64263691-60bb-4559-86ad-0ec935f6357a\") "
Sep 15 19:13:23.987108 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:23.987074    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64263691-60bb-4559-86ad-0ec935f6357a-maas-api-tls\") pod \"64263691-60bb-4559-86ad-0ec935f6357a\" (UID: \"64263691-60bb-4559-86ad-0ec935f6357a\") "
Sep 15 19:13:23.987284 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:23.987258    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64263691-60bb-4559-86ad-0ec935f6357a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "64263691-60bb-4559-86ad-0ec935f6357a" (UID: "64263691-60bb-4559-86ad-0ec935f6357a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:13:23.987749 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:23.987725    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64263691-60bb-4559-86ad-0ec935f6357a-openshift-service-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:13:23.989381 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:23.989359    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64263691-60bb-4559-86ad-0ec935f6357a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "64263691-60bb-4559-86ad-0ec935f6357a" (UID: "64263691-60bb-4559-86ad-0ec935f6357a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:13:23.989473 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:23.989387    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64263691-60bb-4559-86ad-0ec935f6357a-kube-api-access-vbvwh" (OuterVolumeSpecName: "kube-api-access-vbvwh") pod "64263691-60bb-4559-86ad-0ec935f6357a" (UID: "64263691-60bb-4559-86ad-0ec935f6357a"). InnerVolumeSpecName "kube-api-access-vbvwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:13:23.989906 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:23.989873    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64263691-60bb-4559-86ad-0ec935f6357a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "64263691-60bb-4559-86ad-0ec935f6357a" (UID: "64263691-60bb-4559-86ad-0ec935f6357a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:13:23.995851 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:23.995829    2555 scope.go:117] "RemoveContainer" containerID="9e0397c61b080ed933b65d86794ad5d37afae08d1bba233b23406cc1a46aa9e0"
Sep 15 19:13:23.996188 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:13:23.996170    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0397c61b080ed933b65d86794ad5d37afae08d1bba233b23406cc1a46aa9e0\": container with ID starting with 9e0397c61b080ed933b65d86794ad5d37afae08d1bba233b23406cc1a46aa9e0 not found: ID does not exist" containerID="9e0397c61b080ed933b65d86794ad5d37afae08d1bba233b23406cc1a46aa9e0"
Sep 15 19:13:23.996241 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:23.996197    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0397c61b080ed933b65d86794ad5d37afae08d1bba233b23406cc1a46aa9e0"} err="failed to get container status \"9e0397c61b080ed933b65d86794ad5d37afae08d1bba233b23406cc1a46aa9e0\": rpc error: code = NotFound desc = could not find container \"9e0397c61b080ed933b65d86794ad5d37afae08d1bba233b23406cc1a46aa9e0\": container with ID starting with 9e0397c61b080ed933b65d86794ad5d37afae08d1bba233b23406cc1a46aa9e0 not found: ID does not exist"
Sep 15 19:13:24.089035 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:24.088998    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64263691-60bb-4559-86ad-0ec935f6357a-metrics-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:13:24.089035 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:24.089029    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64263691-60bb-4559-86ad-0ec935f6357a-maas-api-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:13:24.089035 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:24.089038    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbvwh\" (UniqueName: \"kubernetes.io/projected/64263691-60bb-4559-86ad-0ec935f6357a-kube-api-access-vbvwh\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:13:24.307072 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:24.307037    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"]
Sep 15 19:13:24.308986 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:24.308965    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d2e6cc70-6d5d4c8767-755fh"]
Sep 15 19:13:25.040669 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:25.040641    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64263691-60bb-4559-86ad-0ec935f6357a" path="/var/lib/kubelet/pods/64263691-60bb-4559-86ad-0ec935f6357a/volumes"
Sep 15 19:13:32.242265 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:32.242209    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"]
Sep 15 19:13:47.262348 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.262281    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8" podUID="03e11035-89d4-42ed-9a1b-13d343f3ce7a" containerName="main" containerID="cri-o://02e2f4a0fde4b26553686dafdc0e66148384fbafb390e649e36d777c3ebcd5a7" gracePeriod=45
Sep 15 19:13:47.516637 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.516560    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:13:47.635069 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.635034    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-tmp-dir\") pod \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") "
Sep 15 19:13:47.635283 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.635122    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-model-cache\") pod \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") "
Sep 15 19:13:47.635283 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.635165    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxzsv\" (UniqueName: \"kubernetes.io/projected/03e11035-89d4-42ed-9a1b-13d343f3ce7a-kube-api-access-mxzsv\") pod \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") "
Sep 15 19:13:47.635283 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.635216    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e11035-89d4-42ed-9a1b-13d343f3ce7a-tls-certs\") pod \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") "
Sep 15 19:13:47.635283 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.635246    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-dshm\") pod \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") "
Sep 15 19:13:47.635495 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.635322    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-home\") pod \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\" (UID: \"03e11035-89d4-42ed-9a1b-13d343f3ce7a\") "
Sep 15 19:13:47.635495 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.635354    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "03e11035-89d4-42ed-9a1b-13d343f3ce7a" (UID: "03e11035-89d4-42ed-9a1b-13d343f3ce7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:13:47.635495 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.635364    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-model-cache" (OuterVolumeSpecName: "model-cache") pod "03e11035-89d4-42ed-9a1b-13d343f3ce7a" (UID: "03e11035-89d4-42ed-9a1b-13d343f3ce7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:13:47.635682 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.635594    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-home" (OuterVolumeSpecName: "home") pod "03e11035-89d4-42ed-9a1b-13d343f3ce7a" (UID: "03e11035-89d4-42ed-9a1b-13d343f3ce7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:13:47.635743 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.635698    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-tmp-dir\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:13:47.635743 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.635717    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-model-cache\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:13:47.635743 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.635731    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-home\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:13:47.637521 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.637489    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-dshm" (OuterVolumeSpecName: "dshm") pod "03e11035-89d4-42ed-9a1b-13d343f3ce7a" (UID: "03e11035-89d4-42ed-9a1b-13d343f3ce7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:13:47.637521 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.637507    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e11035-89d4-42ed-9a1b-13d343f3ce7a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03e11035-89d4-42ed-9a1b-13d343f3ce7a" (UID: "03e11035-89d4-42ed-9a1b-13d343f3ce7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:13:47.637718 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.637548    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e11035-89d4-42ed-9a1b-13d343f3ce7a-kube-api-access-mxzsv" (OuterVolumeSpecName: "kube-api-access-mxzsv") pod "03e11035-89d4-42ed-9a1b-13d343f3ce7a" (UID: "03e11035-89d4-42ed-9a1b-13d343f3ce7a"). InnerVolumeSpecName "kube-api-access-mxzsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:13:47.737050 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.737005    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxzsv\" (UniqueName: \"kubernetes.io/projected/03e11035-89d4-42ed-9a1b-13d343f3ce7a-kube-api-access-mxzsv\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:13:47.737050 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.737044    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e11035-89d4-42ed-9a1b-13d343f3ce7a-tls-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:13:47.737050 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:47.737056    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e11035-89d4-42ed-9a1b-13d343f3ce7a-dshm\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:13:48.114955 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:48.114919    2555 generic.go:356] "Generic (PLEG): container finished" podID="03e11035-89d4-42ed-9a1b-13d343f3ce7a" containerID="02e2f4a0fde4b26553686dafdc0e66148384fbafb390e649e36d777c3ebcd5a7" exitCode=0
Sep 15 19:13:48.115133 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:48.114968    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"
Sep 15 19:13:48.115133 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:48.115006    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8" event={"ID":"03e11035-89d4-42ed-9a1b-13d343f3ce7a","Type":"ContainerDied","Data":"02e2f4a0fde4b26553686dafdc0e66148384fbafb390e649e36d777c3ebcd5a7"}
Sep 15 19:13:48.115133 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:48.115056    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8" event={"ID":"03e11035-89d4-42ed-9a1b-13d343f3ce7a","Type":"ContainerDied","Data":"2d6145075be6ced5c570d41f1a981e5be6a16af5b419b13ecba5da00dde0aab2"}
Sep 15 19:13:48.115133 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:48.115078    2555 scope.go:117] "RemoveContainer" containerID="02e2f4a0fde4b26553686dafdc0e66148384fbafb390e649e36d777c3ebcd5a7"
Sep 15 19:13:48.125044 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:48.125023    2555 scope.go:117] "RemoveContainer" containerID="02e2f4a0fde4b26553686dafdc0e66148384fbafb390e649e36d777c3ebcd5a7"
Sep 15 19:13:48.125327 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:13:48.125310    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e2f4a0fde4b26553686dafdc0e66148384fbafb390e649e36d777c3ebcd5a7\": container with ID starting with 02e2f4a0fde4b26553686dafdc0e66148384fbafb390e649e36d777c3ebcd5a7 not found: ID does not exist" containerID="02e2f4a0fde4b26553686dafdc0e66148384fbafb390e649e36d777c3ebcd5a7"
Sep 15 19:13:48.125378 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:48.125343    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e2f4a0fde4b26553686dafdc0e66148384fbafb390e649e36d777c3ebcd5a7"} err="failed to get container status \"02e2f4a0fde4b26553686dafdc0e66148384fbafb390e649e36d777c3ebcd5a7\": rpc error: code = NotFound desc = could not find container \"02e2f4a0fde4b26553686dafdc0e66148384fbafb390e649e36d777c3ebcd5a7\": container with ID starting with 02e2f4a0fde4b26553686dafdc0e66148384fbafb390e649e36d777c3ebcd5a7 not found: ID does not exist"
Sep 15 19:13:48.136123 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:48.136098    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"]
Sep 15 19:13:48.139746 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:48.139720    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d2e6cc70/e2e-lifecycle-model-d2e6cc70-kserve-578c76755-v9pq8"]
Sep 15 19:13:49.040693 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:49.040663    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03e11035-89d4-42ed-9a1b-13d343f3ce7a" path="/var/lib/kubelet/pods/03e11035-89d4-42ed-9a1b-13d343f3ce7a/volumes"
Sep 15 19:13:51.978861 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:51.978824    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"]
Sep 15 19:13:51.983640 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:51.983590    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"]
Sep 15 19:13:52.009624 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:13:52.009586    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"]
Sep 15 19:14:07.000304 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.000256    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn" podUID="4e2aaba7-d35b-48a9-9927-beb5a9a35ad9" containerName="main" containerID="cri-o://a40f7f496b6233e2360a166f604d19c15afcc7765dd9ae15f7aafc3c67b1aede" gracePeriod=45
Sep 15 19:14:07.002512 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.002485    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v" podUID="26830a72-a1ff-42e4-9693-d40ca366462e" containerName="main" containerID="cri-o://5c50f900675c627a0f5f2246de68e723894c7c77b01f13dd3cfa06182aaef5ab" gracePeriod=45
Sep 15 19:14:07.029048 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.028994    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g" podUID="a312e92a-cbd1-4cea-add8-eb83f2d48a71" containerName="main" containerID="cri-o://8cb88b872b15e9b46e7820d202b17dcf1ca7eb02f064349c78ad934b1dadb803" gracePeriod=45
Sep 15 19:14:07.223106 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.223077    2555 generic.go:356] "Generic (PLEG): container finished" podID="26830a72-a1ff-42e4-9693-d40ca366462e" containerID="5c50f900675c627a0f5f2246de68e723894c7c77b01f13dd3cfa06182aaef5ab" exitCode=0
Sep 15 19:14:07.223246 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.223155    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v" event={"ID":"26830a72-a1ff-42e4-9693-d40ca366462e","Type":"ContainerDied","Data":"5c50f900675c627a0f5f2246de68e723894c7c77b01f13dd3cfa06182aaef5ab"}
Sep 15 19:14:07.225112 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.225091    2555 generic.go:356] "Generic (PLEG): container finished" podID="a312e92a-cbd1-4cea-add8-eb83f2d48a71" containerID="8cb88b872b15e9b46e7820d202b17dcf1ca7eb02f064349c78ad934b1dadb803" exitCode=0
Sep 15 19:14:07.225238 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.225169    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g" event={"ID":"a312e92a-cbd1-4cea-add8-eb83f2d48a71","Type":"ContainerDied","Data":"8cb88b872b15e9b46e7820d202b17dcf1ca7eb02f064349c78ad934b1dadb803"}
Sep 15 19:14:07.226844 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.226823    2555 generic.go:356] "Generic (PLEG): container finished" podID="4e2aaba7-d35b-48a9-9927-beb5a9a35ad9" containerID="a40f7f496b6233e2360a166f604d19c15afcc7765dd9ae15f7aafc3c67b1aede" exitCode=0
Sep 15 19:14:07.226920 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.226866    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn" event={"ID":"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9","Type":"ContainerDied","Data":"a40f7f496b6233e2360a166f604d19c15afcc7765dd9ae15f7aafc3c67b1aede"}
Sep 15 19:14:07.295176 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.295150    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:14:07.303946 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.303923    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:14:07.318064 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.318028    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:14:07.336546 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.336473    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-tmp-dir\") pod \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") "
Sep 15 19:14:07.336546 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.336515    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-model-cache\") pod \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") "
Sep 15 19:14:07.336546 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.336547    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-home\") pod \"26830a72-a1ff-42e4-9693-d40ca366462e\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") "
Sep 15 19:14:07.336860 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.336588    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-home\") pod \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") "
Sep 15 19:14:07.336860 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.336663    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-dshm\") pod \"26830a72-a1ff-42e4-9693-d40ca366462e\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") "
Sep 15 19:14:07.336860 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.336704    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-tmp-dir\") pod \"26830a72-a1ff-42e4-9693-d40ca366462e\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") "
Sep 15 19:14:07.336860 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.336760    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-tls-certs\") pod \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") "
Sep 15 19:14:07.336860 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.336793    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-model-cache\") pod \"26830a72-a1ff-42e4-9693-d40ca366462e\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") "
Sep 15 19:14:07.336860 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.336818    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqsc6\" (UniqueName: \"kubernetes.io/projected/26830a72-a1ff-42e4-9693-d40ca366462e-kube-api-access-kqsc6\") pod \"26830a72-a1ff-42e4-9693-d40ca366462e\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") "
Sep 15 19:14:07.337187 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.336835    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e2aaba7-d35b-48a9-9927-beb5a9a35ad9" (UID: "4e2aaba7-d35b-48a9-9927-beb5a9a35ad9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.337187 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.336884    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26830a72-a1ff-42e4-9693-d40ca366462e-tls-certs\") pod \"26830a72-a1ff-42e4-9693-d40ca366462e\" (UID: \"26830a72-a1ff-42e4-9693-d40ca366462e\") "
Sep 15 19:14:07.337187 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.336922    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-home" (OuterVolumeSpecName: "home") pod "4e2aaba7-d35b-48a9-9927-beb5a9a35ad9" (UID: "4e2aaba7-d35b-48a9-9927-beb5a9a35ad9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.337187 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.337017    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e2aaba7-d35b-48a9-9927-beb5a9a35ad9" (UID: "4e2aaba7-d35b-48a9-9927-beb5a9a35ad9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.337187 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.337136    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "26830a72-a1ff-42e4-9693-d40ca366462e" (UID: "26830a72-a1ff-42e4-9693-d40ca366462e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.337187 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.337152    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-dshm\") pod \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") "
Sep 15 19:14:07.337494 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.337201    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sz24\" (UniqueName: \"kubernetes.io/projected/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-kube-api-access-2sz24\") pod \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\" (UID: \"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9\") "
Sep 15 19:14:07.337494 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.337227    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-home" (OuterVolumeSpecName: "home") pod "26830a72-a1ff-42e4-9693-d40ca366462e" (UID: "26830a72-a1ff-42e4-9693-d40ca366462e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.337494 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.337361    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-model-cache" (OuterVolumeSpecName: "model-cache") pod "26830a72-a1ff-42e4-9693-d40ca366462e" (UID: "26830a72-a1ff-42e4-9693-d40ca366462e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.337811 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.337702    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-model-cache\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.337811 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.337727    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-tmp-dir\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.337811 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.337742    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-model-cache\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.337811 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.337753    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-home\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.337811 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.337765    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-home\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.337811 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.337777    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-tmp-dir\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.339853 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.339815    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-dshm" (OuterVolumeSpecName: "dshm") pod "4e2aaba7-d35b-48a9-9927-beb5a9a35ad9" (UID: "4e2aaba7-d35b-48a9-9927-beb5a9a35ad9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.340401 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.340365    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26830a72-a1ff-42e4-9693-d40ca366462e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26830a72-a1ff-42e4-9693-d40ca366462e" (UID: "26830a72-a1ff-42e4-9693-d40ca366462e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:14:07.340401 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.340270    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-dshm" (OuterVolumeSpecName: "dshm") pod "26830a72-a1ff-42e4-9693-d40ca366462e" (UID: "26830a72-a1ff-42e4-9693-d40ca366462e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.340567 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.340415    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-kube-api-access-2sz24" (OuterVolumeSpecName: "kube-api-access-2sz24") pod "4e2aaba7-d35b-48a9-9927-beb5a9a35ad9" (UID: "4e2aaba7-d35b-48a9-9927-beb5a9a35ad9"). InnerVolumeSpecName "kube-api-access-2sz24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:14:07.340567 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.340498    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e2aaba7-d35b-48a9-9927-beb5a9a35ad9" (UID: "4e2aaba7-d35b-48a9-9927-beb5a9a35ad9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:14:07.341051 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.341025    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26830a72-a1ff-42e4-9693-d40ca366462e-kube-api-access-kqsc6" (OuterVolumeSpecName: "kube-api-access-kqsc6") pod "26830a72-a1ff-42e4-9693-d40ca366462e" (UID: "26830a72-a1ff-42e4-9693-d40ca366462e"). InnerVolumeSpecName "kube-api-access-kqsc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:14:07.439048 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.439018    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-model-cache\") pod \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") "
Sep 15 19:14:07.439221 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.439068    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6lk7\" (UniqueName: \"kubernetes.io/projected/a312e92a-cbd1-4cea-add8-eb83f2d48a71-kube-api-access-c6lk7\") pod \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") "
Sep 15 19:14:07.439221 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.439130    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a312e92a-cbd1-4cea-add8-eb83f2d48a71-tls-certs\") pod \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") "
Sep 15 19:14:07.439221 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.439178    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-tmp-dir\") pod \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") "
Sep 15 19:14:07.439221 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.439206    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-dshm\") pod \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") "
Sep 15 19:14:07.439438 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.439268    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-home\") pod \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\" (UID: \"a312e92a-cbd1-4cea-add8-eb83f2d48a71\") "
Sep 15 19:14:07.439438 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.439302    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-model-cache" (OuterVolumeSpecName: "model-cache") pod "a312e92a-cbd1-4cea-add8-eb83f2d48a71" (UID: "a312e92a-cbd1-4cea-add8-eb83f2d48a71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.439548 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.439503    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a312e92a-cbd1-4cea-add8-eb83f2d48a71" (UID: "a312e92a-cbd1-4cea-add8-eb83f2d48a71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.439548 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.439530    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-home" (OuterVolumeSpecName: "home") pod "a312e92a-cbd1-4cea-add8-eb83f2d48a71" (UID: "a312e92a-cbd1-4cea-add8-eb83f2d48a71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.439680 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.439651    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-model-cache\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.439680 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.439671    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26830a72-a1ff-42e4-9693-d40ca366462e-dshm\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.439764 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.439684    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-tls-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.439764 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.439697    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqsc6\" (UniqueName: \"kubernetes.io/projected/26830a72-a1ff-42e4-9693-d40ca366462e-kube-api-access-kqsc6\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.439764 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.439709    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-tmp-dir\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.439764 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.439722    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26830a72-a1ff-42e4-9693-d40ca366462e-tls-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.439764 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.439735    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-dshm\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.439764 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.439749    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sz24\" (UniqueName: \"kubernetes.io/projected/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9-kube-api-access-2sz24\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.441259 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.441239    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a312e92a-cbd1-4cea-add8-eb83f2d48a71-kube-api-access-c6lk7" (OuterVolumeSpecName: "kube-api-access-c6lk7") pod "a312e92a-cbd1-4cea-add8-eb83f2d48a71" (UID: "a312e92a-cbd1-4cea-add8-eb83f2d48a71"). InnerVolumeSpecName "kube-api-access-c6lk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:14:07.441691 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.441668    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-dshm" (OuterVolumeSpecName: "dshm") pod "a312e92a-cbd1-4cea-add8-eb83f2d48a71" (UID: "a312e92a-cbd1-4cea-add8-eb83f2d48a71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:07.441744 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.441671    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a312e92a-cbd1-4cea-add8-eb83f2d48a71-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a312e92a-cbd1-4cea-add8-eb83f2d48a71" (UID: "a312e92a-cbd1-4cea-add8-eb83f2d48a71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:14:07.540973 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.540901    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-dshm\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.540973 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.540927    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a312e92a-cbd1-4cea-add8-eb83f2d48a71-home\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.540973 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.540936    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6lk7\" (UniqueName: \"kubernetes.io/projected/a312e92a-cbd1-4cea-add8-eb83f2d48a71-kube-api-access-c6lk7\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:14:07.540973 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:07.540947    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a312e92a-cbd1-4cea-add8-eb83f2d48a71-tls-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:14:08.233573 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:08.233543    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"
Sep 15 19:14:08.234000 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:08.233582    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v" event={"ID":"26830a72-a1ff-42e4-9693-d40ca366462e","Type":"ContainerDied","Data":"c97b4d0070608ced5db145d897715710eac6912d99b9f0132469d97e9381f4f5"}
Sep 15 19:14:08.234000 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:08.233644    2555 scope.go:117] "RemoveContainer" containerID="5c50f900675c627a0f5f2246de68e723894c7c77b01f13dd3cfa06182aaef5ab"
Sep 15 19:14:08.235053 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:08.235024    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"
Sep 15 19:14:08.235161 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:08.235058    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g" event={"ID":"a312e92a-cbd1-4cea-add8-eb83f2d48a71","Type":"ContainerDied","Data":"1bb42a892eb7dec2b9357447e12aed0a8fbe6b1c379cf9cf7122051ad6072307"}
Sep 15 19:14:08.236809 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:08.236783    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn" event={"ID":"4e2aaba7-d35b-48a9-9927-beb5a9a35ad9","Type":"ContainerDied","Data":"670ff726097506825552c6ff9dfd1fed0a6faa805ced294c2771c415bb8807c4"}
Sep 15 19:14:08.236944 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:08.236826    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"
Sep 15 19:14:08.244386 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:08.244362    2555 scope.go:117] "RemoveContainer" containerID="8cb88b872b15e9b46e7820d202b17dcf1ca7eb02f064349c78ad934b1dadb803"
Sep 15 19:14:08.252599 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:08.252580    2555 scope.go:117] "RemoveContainer" containerID="a40f7f496b6233e2360a166f604d19c15afcc7765dd9ae15f7aafc3c67b1aede"
Sep 15 19:14:08.256090 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:08.256068    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"]
Sep 15 19:14:08.261785 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:08.261762    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/distinct-w0-74ef1f-kserve-5487894456-9s92v"]
Sep 15 19:14:08.270942 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:08.270891    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"]
Sep 15 19:14:08.274198 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:08.274178    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g"]
Sep 15 19:14:08.284216 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:08.284194    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"]
Sep 15 19:14:08.285955 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:08.285937    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-74ef1f/distinct-2-w0-74ef1f-kserve-d644f9555-5xbrn"]
Sep 15 19:14:09.040034 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:09.039997    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26830a72-a1ff-42e4-9693-d40ca366462e" path="/var/lib/kubelet/pods/26830a72-a1ff-42e4-9693-d40ca366462e/volumes"
Sep 15 19:14:09.040406 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:09.040393    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e2aaba7-d35b-48a9-9927-beb5a9a35ad9" path="/var/lib/kubelet/pods/4e2aaba7-d35b-48a9-9927-beb5a9a35ad9/volumes"
Sep 15 19:14:09.040763 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:09.040747    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a312e92a-cbd1-4cea-add8-eb83f2d48a71" path="/var/lib/kubelet/pods/a312e92a-cbd1-4cea-add8-eb83f2d48a71/volumes"
Sep 15 19:14:12.275300 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:12.275250    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-74ef1f/simulator-w0-74ef1f-kserve-5c7df67fb4-jqc6g" podUID="a312e92a-cbd1-4cea-add8-eb83f2d48a71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 15 19:14:56.017393 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.017310    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8927-ec29e7"]
Sep 15 19:14:56.017889 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.017730    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64263691-60bb-4559-86ad-0ec935f6357a" containerName="maas-api"
Sep 15 19:14:56.017889 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.017742    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="64263691-60bb-4559-86ad-0ec935f6357a" containerName="maas-api"
Sep 15 19:14:56.017889 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.017755    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26830a72-a1ff-42e4-9693-d40ca366462e" containerName="main"
Sep 15 19:14:56.017889 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.017761    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="26830a72-a1ff-42e4-9693-d40ca366462e" containerName="main"
Sep 15 19:14:56.017889 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.017775    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a312e92a-cbd1-4cea-add8-eb83f2d48a71" containerName="main"
Sep 15 19:14:56.017889 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.017780    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a312e92a-cbd1-4cea-add8-eb83f2d48a71" containerName="main"
Sep 15 19:14:56.017889 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.017787    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2aaba7-d35b-48a9-9927-beb5a9a35ad9" containerName="main"
Sep 15 19:14:56.017889 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.017791    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2aaba7-d35b-48a9-9927-beb5a9a35ad9" containerName="main"
Sep 15 19:14:56.017889 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.017807    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03e11035-89d4-42ed-9a1b-13d343f3ce7a" containerName="main"
Sep 15 19:14:56.017889 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.017812    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="03e11035-89d4-42ed-9a1b-13d343f3ce7a" containerName="main"
Sep 15 19:14:56.017889 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.017873    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a312e92a-cbd1-4cea-add8-eb83f2d48a71" containerName="main"
Sep 15 19:14:56.017889 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.017880    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="26830a72-a1ff-42e4-9693-d40ca366462e" containerName="main"
Sep 15 19:14:56.017889 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.017887    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e2aaba7-d35b-48a9-9927-beb5a9a35ad9" containerName="main"
Sep 15 19:14:56.017889 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.017894    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="64263691-60bb-4559-86ad-0ec935f6357a" containerName="maas-api"
Sep 15 19:14:56.018331 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.017899    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="03e11035-89d4-42ed-9a1b-13d343f3ce7a" containerName="main"
Sep 15 19:14:56.021161 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.021138    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8927-ec29e7"
Sep 15 19:14:56.023181 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.023159    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zrrnn\""
Sep 15 19:14:56.024306 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.024278    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8927-ec29e7"]
Sep 15 19:14:56.115484 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.115446    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzckn\" (UniqueName: \"kubernetes.io/projected/bade0a68-100f-49d7-8ba7-f5042a4b3895-kube-api-access-fzckn\") pod \"test-curl-8927-ec29e7\" (UID: \"bade0a68-100f-49d7-8ba7-f5042a4b3895\") " pod="opendatahub/test-curl-8927-ec29e7"
Sep 15 19:14:56.216213 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.216168    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzckn\" (UniqueName: \"kubernetes.io/projected/bade0a68-100f-49d7-8ba7-f5042a4b3895-kube-api-access-fzckn\") pod \"test-curl-8927-ec29e7\" (UID: \"bade0a68-100f-49d7-8ba7-f5042a4b3895\") " pod="opendatahub/test-curl-8927-ec29e7"
Sep 15 19:14:56.224275 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.224250    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzckn\" (UniqueName: \"kubernetes.io/projected/bade0a68-100f-49d7-8ba7-f5042a4b3895-kube-api-access-fzckn\") pod \"test-curl-8927-ec29e7\" (UID: \"bade0a68-100f-49d7-8ba7-f5042a4b3895\") " pod="opendatahub/test-curl-8927-ec29e7"
Sep 15 19:14:56.331878 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.331801    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8927-ec29e7"
Sep 15 19:14:56.600622 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:56.600581    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8927-ec29e7"]
Sep 15 19:14:56.602578 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:14:56.602552    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbade0a68_100f_49d7_8ba7_f5042a4b3895.slice/crio-1ddc2cce2757602c6e88d68ecd4e7275daeb75e4cfcc147bffeb53d697c35e1a WatchSource:0}: Error finding container 1ddc2cce2757602c6e88d68ecd4e7275daeb75e4cfcc147bffeb53d697c35e1a: Status 404 returned error can't find the container with id 1ddc2cce2757602c6e88d68ecd4e7275daeb75e4cfcc147bffeb53d697c35e1a
Sep 15 19:14:57.506387 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:57.506353    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8927-ec29e7" event={"ID":"bade0a68-100f-49d7-8ba7-f5042a4b3895","Type":"ContainerStarted","Data":"c867549599fcd3ca77bcb85fce883c18a160e94bd39f3596c79cc2bf9d218d3f"}
Sep 15 19:14:57.506791 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:57.506395    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8927-ec29e7" event={"ID":"bade0a68-100f-49d7-8ba7-f5042a4b3895","Type":"ContainerStarted","Data":"1ddc2cce2757602c6e88d68ecd4e7275daeb75e4cfcc147bffeb53d697c35e1a"}
Sep 15 19:14:57.520367 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:57.520318    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8927-ec29e7" podStartSLOduration=1.520300528 podStartE2EDuration="1.520300528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:14:57.518829205 +0000 UTC m=+1439.006158191" watchObservedRunningTime="2026-09-15 19:14:57.520300528 +0000 UTC m=+1439.007629513"
Sep 15 19:14:58.128492 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:58.128455    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8927-ec29e7"]
Sep 15 19:14:58.131812 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:58.131772    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8927-ec29e7"]
Sep 15 19:14:58.511802 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:14:58.511773    2555 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8927-ec29e7" container="test-curl-8927-ec29e7"
Sep 15 19:14:59.516761 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:14:59.516721    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8927-ec29e7" podUID="bade0a68-100f-49d7-8ba7-f5042a4b3895" containerName="test-curl-8927-ec29e7" containerID="cri-o://c867549599fcd3ca77bcb85fce883c18a160e94bd39f3596c79cc2bf9d218d3f" gracePeriod=2
Sep 15 19:15:00.649841 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:00.649808    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"]
Sep 15 19:15:01.710183 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:01.710159    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8927-ec29e7"
Sep 15 19:15:01.873002 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:01.872906    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzckn\" (UniqueName: \"kubernetes.io/projected/bade0a68-100f-49d7-8ba7-f5042a4b3895-kube-api-access-fzckn\") pod \"bade0a68-100f-49d7-8ba7-f5042a4b3895\" (UID: \"bade0a68-100f-49d7-8ba7-f5042a4b3895\") "
Sep 15 19:15:01.875048 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:01.875014    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bade0a68-100f-49d7-8ba7-f5042a4b3895-kube-api-access-fzckn" (OuterVolumeSpecName: "kube-api-access-fzckn") pod "bade0a68-100f-49d7-8ba7-f5042a4b3895" (UID: "bade0a68-100f-49d7-8ba7-f5042a4b3895"). InnerVolumeSpecName "kube-api-access-fzckn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:15:01.974640 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:01.974568    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzckn\" (UniqueName: \"kubernetes.io/projected/bade0a68-100f-49d7-8ba7-f5042a4b3895-kube-api-access-fzckn\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:15:02.533796 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:02.533763    2555 generic.go:356] "Generic (PLEG): container finished" podID="bade0a68-100f-49d7-8ba7-f5042a4b3895" containerID="c867549599fcd3ca77bcb85fce883c18a160e94bd39f3596c79cc2bf9d218d3f" exitCode=137
Sep 15 19:15:02.533970 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:02.533803    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8927-ec29e7"
Sep 15 19:15:02.533970 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:02.533859    2555 scope.go:117] "RemoveContainer" containerID="c867549599fcd3ca77bcb85fce883c18a160e94bd39f3596c79cc2bf9d218d3f"
Sep 15 19:15:02.543244 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:02.543225    2555 scope.go:117] "RemoveContainer" containerID="c867549599fcd3ca77bcb85fce883c18a160e94bd39f3596c79cc2bf9d218d3f"
Sep 15 19:15:02.543499 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:15:02.543481    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c867549599fcd3ca77bcb85fce883c18a160e94bd39f3596c79cc2bf9d218d3f\": container with ID starting with c867549599fcd3ca77bcb85fce883c18a160e94bd39f3596c79cc2bf9d218d3f not found: ID does not exist" containerID="c867549599fcd3ca77bcb85fce883c18a160e94bd39f3596c79cc2bf9d218d3f"
Sep 15 19:15:02.543544 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:02.543507    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c867549599fcd3ca77bcb85fce883c18a160e94bd39f3596c79cc2bf9d218d3f"} err="failed to get container status \"c867549599fcd3ca77bcb85fce883c18a160e94bd39f3596c79cc2bf9d218d3f\": rpc error: code = NotFound desc = could not find container \"c867549599fcd3ca77bcb85fce883c18a160e94bd39f3596c79cc2bf9d218d3f\": container with ID starting with c867549599fcd3ca77bcb85fce883c18a160e94bd39f3596c79cc2bf9d218d3f not found: ID does not exist"
Sep 15 19:15:02.544600 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:15:02.544583    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8927-ec29e7\" is forbidden: User \"system:node:ip-10-0-132-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-28.ec2.internal' and this object" podUID="bade0a68-100f-49d7-8ba7-f5042a4b3895" pod="opendatahub/test-curl-8927-ec29e7"
Sep 15 19:15:03.041700 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:03.041664    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bade0a68-100f-49d7-8ba7-f5042a4b3895" path="/var/lib/kubelet/pods/bade0a68-100f-49d7-8ba7-f5042a4b3895/volumes"
Sep 15 19:15:15.670111 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:15.670046    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c" podUID="33731e0d-0c55-4977-b8a0-5f8befa9a27a" containerName="main" containerID="cri-o://14ad6e5521e98479373e4a54ffc14b87169561436439dd0dd366dd5931fe9a9b" gracePeriod=45
Sep 15 19:15:15.924032 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:15.923966    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:15:15.963362 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:15.963328    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"]
Sep 15 19:15:15.964044 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:15.964019    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33731e0d-0c55-4977-b8a0-5f8befa9a27a" containerName="main"
Sep 15 19:15:15.964044 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:15.964045    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="33731e0d-0c55-4977-b8a0-5f8befa9a27a" containerName="main"
Sep 15 19:15:15.964193 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:15.964064    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bade0a68-100f-49d7-8ba7-f5042a4b3895" containerName="test-curl-8927-ec29e7"
Sep 15 19:15:15.964193 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:15.964073    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bade0a68-100f-49d7-8ba7-f5042a4b3895" containerName="test-curl-8927-ec29e7"
Sep 15 19:15:15.964193 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:15.964188    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bade0a68-100f-49d7-8ba7-f5042a4b3895" containerName="test-curl-8927-ec29e7"
Sep 15 19:15:15.964291 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:15.964206    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="33731e0d-0c55-4977-b8a0-5f8befa9a27a" containerName="main"
Sep 15 19:15:15.967938 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:15.967916    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"
Sep 15 19:15:15.970123 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:15.970094    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-910e11-serving-cert\""
Sep 15 19:15:15.977416 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:15.977350    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"]
Sep 15 19:15:16.002294 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.002258    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33731e0d-0c55-4977-b8a0-5f8befa9a27a-tls-certs\") pod \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") "
Sep 15 19:15:16.002450 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.002305    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvpxc\" (UniqueName: \"kubernetes.io/projected/33731e0d-0c55-4977-b8a0-5f8befa9a27a-kube-api-access-vvpxc\") pod \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") "
Sep 15 19:15:16.002450 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.002346    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-home\") pod \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") "
Sep 15 19:15:16.002587 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.002443    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-model-cache\") pod \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") "
Sep 15 19:15:16.002587 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.002485    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-tmp-dir\") pod \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") "
Sep 15 19:15:16.002587 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.002536    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-dshm\") pod \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\" (UID: \"33731e0d-0c55-4977-b8a0-5f8befa9a27a\") "
Sep 15 19:15:16.002587 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.002566    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-home" (OuterVolumeSpecName: "home") pod "33731e0d-0c55-4977-b8a0-5f8befa9a27a" (UID: "33731e0d-0c55-4977-b8a0-5f8befa9a27a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:16.002800 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.002699    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-model-cache" (OuterVolumeSpecName: "model-cache") pod "33731e0d-0c55-4977-b8a0-5f8befa9a27a" (UID: "33731e0d-0c55-4977-b8a0-5f8befa9a27a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:16.002855 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.002790    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33731e0d-0c55-4977-b8a0-5f8befa9a27a" (UID: "33731e0d-0c55-4977-b8a0-5f8befa9a27a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:16.002855 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.002835    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d9a2f8-04b4-4728-b786-68974d219708-openshift-service-ca\") pod \"maas-api-e2e-api-a-910e11-69795bd8f5-hw287\" (UID: \"e0d9a2f8-04b4-4728-b786-68974d219708\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"
Sep 15 19:15:16.002966 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.002899    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0d9a2f8-04b4-4728-b786-68974d219708-maas-api-tls\") pod \"maas-api-e2e-api-a-910e11-69795bd8f5-hw287\" (UID: \"e0d9a2f8-04b4-4728-b786-68974d219708\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"
Sep 15 19:15:16.002966 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.002955    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fbvd\" (UniqueName: \"kubernetes.io/projected/e0d9a2f8-04b4-4728-b786-68974d219708-kube-api-access-6fbvd\") pod \"maas-api-e2e-api-a-910e11-69795bd8f5-hw287\" (UID: \"e0d9a2f8-04b4-4728-b786-68974d219708\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"
Sep 15 19:15:16.003079 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.003056    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0d9a2f8-04b4-4728-b786-68974d219708-metrics-certs\") pod \"maas-api-e2e-api-a-910e11-69795bd8f5-hw287\" (UID: \"e0d9a2f8-04b4-4728-b786-68974d219708\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"
Sep 15 19:15:16.003160 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.003144    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-home\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:15:16.003205 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.003165    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-model-cache\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:15:16.003205 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.003179    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-tmp-dir\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:15:16.004636 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.004592    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-dshm" (OuterVolumeSpecName: "dshm") pod "33731e0d-0c55-4977-b8a0-5f8befa9a27a" (UID: "33731e0d-0c55-4977-b8a0-5f8befa9a27a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:16.004750 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.004634    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33731e0d-0c55-4977-b8a0-5f8befa9a27a-kube-api-access-vvpxc" (OuterVolumeSpecName: "kube-api-access-vvpxc") pod "33731e0d-0c55-4977-b8a0-5f8befa9a27a" (UID: "33731e0d-0c55-4977-b8a0-5f8befa9a27a"). InnerVolumeSpecName "kube-api-access-vvpxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:15:16.004750 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.004643    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33731e0d-0c55-4977-b8a0-5f8befa9a27a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33731e0d-0c55-4977-b8a0-5f8befa9a27a" (UID: "33731e0d-0c55-4977-b8a0-5f8befa9a27a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:15:16.103847 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.103801    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0d9a2f8-04b4-4728-b786-68974d219708-metrics-certs\") pod \"maas-api-e2e-api-a-910e11-69795bd8f5-hw287\" (UID: \"e0d9a2f8-04b4-4728-b786-68974d219708\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"
Sep 15 19:15:16.104037 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.103903    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d9a2f8-04b4-4728-b786-68974d219708-openshift-service-ca\") pod \"maas-api-e2e-api-a-910e11-69795bd8f5-hw287\" (UID: \"e0d9a2f8-04b4-4728-b786-68974d219708\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"
Sep 15 19:15:16.104037 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.103952    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0d9a2f8-04b4-4728-b786-68974d219708-maas-api-tls\") pod \"maas-api-e2e-api-a-910e11-69795bd8f5-hw287\" (UID: \"e0d9a2f8-04b4-4728-b786-68974d219708\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"
Sep 15 19:15:16.104037 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.103990    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fbvd\" (UniqueName: \"kubernetes.io/projected/e0d9a2f8-04b4-4728-b786-68974d219708-kube-api-access-6fbvd\") pod \"maas-api-e2e-api-a-910e11-69795bd8f5-hw287\" (UID: \"e0d9a2f8-04b4-4728-b786-68974d219708\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"
Sep 15 19:15:16.104156 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.104044    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33731e0d-0c55-4977-b8a0-5f8befa9a27a-tls-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:15:16.104156 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.104060    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvpxc\" (UniqueName: \"kubernetes.io/projected/33731e0d-0c55-4977-b8a0-5f8befa9a27a-kube-api-access-vvpxc\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:15:16.104156 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.104073    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33731e0d-0c55-4977-b8a0-5f8befa9a27a-dshm\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:15:16.104156 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:15:16.104110    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-serving-cert: secret "maas-api-e2e-api-a-910e11-serving-cert" not found
Sep 15 19:15:16.104306 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:15:16.104173    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0d9a2f8-04b4-4728-b786-68974d219708-maas-api-tls podName:e0d9a2f8-04b4-4728-b786-68974d219708 nodeName:}" failed. No retries permitted until 2026-09-15 19:15:16.604158299 +0000 UTC m=+1458.091487262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0d9a2f8-04b4-4728-b786-68974d219708-maas-api-tls") pod "maas-api-e2e-api-a-910e11-69795bd8f5-hw287" (UID: "e0d9a2f8-04b4-4728-b786-68974d219708") : secret "maas-api-e2e-api-a-910e11-serving-cert" not found
Sep 15 19:15:16.104645 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.104625    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d9a2f8-04b4-4728-b786-68974d219708-openshift-service-ca\") pod \"maas-api-e2e-api-a-910e11-69795bd8f5-hw287\" (UID: \"e0d9a2f8-04b4-4728-b786-68974d219708\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"
Sep 15 19:15:16.106264 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.106245    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0d9a2f8-04b4-4728-b786-68974d219708-metrics-certs\") pod \"maas-api-e2e-api-a-910e11-69795bd8f5-hw287\" (UID: \"e0d9a2f8-04b4-4728-b786-68974d219708\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"
Sep 15 19:15:16.112065 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.112049    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fbvd\" (UniqueName: \"kubernetes.io/projected/e0d9a2f8-04b4-4728-b786-68974d219708-kube-api-access-6fbvd\") pod \"maas-api-e2e-api-a-910e11-69795bd8f5-hw287\" (UID: \"e0d9a2f8-04b4-4728-b786-68974d219708\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"
Sep 15 19:15:16.608107 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.608074    2555 generic.go:356] "Generic (PLEG): container finished" podID="33731e0d-0c55-4977-b8a0-5f8befa9a27a" containerID="14ad6e5521e98479373e4a54ffc14b87169561436439dd0dd366dd5931fe9a9b" exitCode=0
Sep 15 19:15:16.608300 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.608115    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"
Sep 15 19:15:16.608300 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.608126    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c" event={"ID":"33731e0d-0c55-4977-b8a0-5f8befa9a27a","Type":"ContainerDied","Data":"14ad6e5521e98479373e4a54ffc14b87169561436439dd0dd366dd5931fe9a9b"}
Sep 15 19:15:16.608300 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.608159    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c" event={"ID":"33731e0d-0c55-4977-b8a0-5f8befa9a27a","Type":"ContainerDied","Data":"19a9ed0895c80f54e4aa0e520f05be3a732c0ec0b8951bbd4ecca3db53052242"}
Sep 15 19:15:16.608300 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.608176    2555 scope.go:117] "RemoveContainer" containerID="14ad6e5521e98479373e4a54ffc14b87169561436439dd0dd366dd5931fe9a9b"
Sep 15 19:15:16.609377 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.609357    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0d9a2f8-04b4-4728-b786-68974d219708-maas-api-tls\") pod \"maas-api-e2e-api-a-910e11-69795bd8f5-hw287\" (UID: \"e0d9a2f8-04b4-4728-b786-68974d219708\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"
Sep 15 19:15:16.611716 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.611694    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0d9a2f8-04b4-4728-b786-68974d219708-maas-api-tls\") pod \"maas-api-e2e-api-a-910e11-69795bd8f5-hw287\" (UID: \"e0d9a2f8-04b4-4728-b786-68974d219708\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"
Sep 15 19:15:16.617836 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.617818    2555 scope.go:117] "RemoveContainer" containerID="14ad6e5521e98479373e4a54ffc14b87169561436439dd0dd366dd5931fe9a9b"
Sep 15 19:15:16.618136 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:15:16.618112    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14ad6e5521e98479373e4a54ffc14b87169561436439dd0dd366dd5931fe9a9b\": container with ID starting with 14ad6e5521e98479373e4a54ffc14b87169561436439dd0dd366dd5931fe9a9b not found: ID does not exist" containerID="14ad6e5521e98479373e4a54ffc14b87169561436439dd0dd366dd5931fe9a9b"
Sep 15 19:15:16.618201 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.618145    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14ad6e5521e98479373e4a54ffc14b87169561436439dd0dd366dd5931fe9a9b"} err="failed to get container status \"14ad6e5521e98479373e4a54ffc14b87169561436439dd0dd366dd5931fe9a9b\": rpc error: code = NotFound desc = could not find container \"14ad6e5521e98479373e4a54ffc14b87169561436439dd0dd366dd5931fe9a9b\": container with ID starting with 14ad6e5521e98479373e4a54ffc14b87169561436439dd0dd366dd5931fe9a9b not found: ID does not exist"
Sep 15 19:15:16.643074 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.643044    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"]
Sep 15 19:15:16.648075 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.648056    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:16.650893 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.650860    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-7073b8\"/\"default-dockercfg-44jww\""
Sep 15 19:15:16.651017 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.650890    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-7073b8\"/\"sub-test-model-7073b8-kserve-self-signed-certs\""
Sep 15 19:15:16.651017 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.650862    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-7073b8\"/\"kube-root-ca.crt\""
Sep 15 19:15:16.651017 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.650911    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-7073b8\"/\"openshift-service-ca.crt\""
Sep 15 19:15:16.673982 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.673949    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"]
Sep 15 19:15:16.676492 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.676464    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"]
Sep 15 19:15:16.695830 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.695798    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-33ff04/auth-test-model-33ff04-kserve-56496684b7-tbs2c"]
Sep 15 19:15:16.710886 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.710859    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e52edd-9587-47c1-b9bc-e8d7a7c30460-tls-certs\") pod \"sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:16.711029 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.710891    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2gkc\" (UniqueName: \"kubernetes.io/projected/88e52edd-9587-47c1-b9bc-e8d7a7c30460-kube-api-access-f2gkc\") pod \"sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:16.711029 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.710913    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-home\") pod \"sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:16.711029 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.710930    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-model-cache\") pod \"sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:16.711029 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.710998    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-tmp-dir\") pod \"sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:16.711162 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.711052    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-dshm\") pod \"sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:16.812722 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.812692    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e52edd-9587-47c1-b9bc-e8d7a7c30460-tls-certs\") pod \"sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:16.812896 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.812735    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2gkc\" (UniqueName: \"kubernetes.io/projected/88e52edd-9587-47c1-b9bc-e8d7a7c30460-kube-api-access-f2gkc\") pod \"sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:16.812896 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.812759    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-home\") pod \"sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:16.812896 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.812779    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-model-cache\") pod \"sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:16.812896 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.812816    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-tmp-dir\") pod \"sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:16.812896 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.812848    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-dshm\") pod \"sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:16.813222 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.813197    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-home\") pod \"sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:16.813294 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.813231    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-model-cache\") pod \"sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:16.813346 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.813290    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-tmp-dir\") pod \"sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:16.815119 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.815098    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-dshm\") pod \"sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:16.815264 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.815248    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e52edd-9587-47c1-b9bc-e8d7a7c30460-tls-certs\") pod \"sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:16.834442 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.834412    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2gkc\" (UniqueName: \"kubernetes.io/projected/88e52edd-9587-47c1-b9bc-e8d7a7c30460-kube-api-access-f2gkc\") pod \"sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") " pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:16.880962 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.880880    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"
Sep 15 19:15:16.957050 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:16.957020    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:17.040484 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:17.040456    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33731e0d-0c55-4977-b8a0-5f8befa9a27a" path="/var/lib/kubelet/pods/33731e0d-0c55-4977-b8a0-5f8befa9a27a/volumes"
Sep 15 19:15:17.157416 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:17.157359    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"]
Sep 15 19:15:17.159853 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:15:17.159821    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d9a2f8_04b4_4728_b786_68974d219708.slice/crio-e042aca5c3b7d86a98a2e24e21eae2721c06d17b00ab43162ab1ab1067c48a19 WatchSource:0}: Error finding container e042aca5c3b7d86a98a2e24e21eae2721c06d17b00ab43162ab1ab1067c48a19: Status 404 returned error can't find the container with id e042aca5c3b7d86a98a2e24e21eae2721c06d17b00ab43162ab1ab1067c48a19
Sep 15 19:15:17.161223 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:17.161206    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:15:17.233315 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:17.233292    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"]
Sep 15 19:15:17.235559 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:15:17.235528    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88e52edd_9587_47c1_b9bc_e8d7a7c30460.slice/crio-40911e460ad49c3626a26c96d68962a9c17bad13f96acbbadf4234f60fc72425 WatchSource:0}: Error finding container 40911e460ad49c3626a26c96d68962a9c17bad13f96acbbadf4234f60fc72425: Status 404 returned error can't find the container with id 40911e460ad49c3626a26c96d68962a9c17bad13f96acbbadf4234f60fc72425
Sep 15 19:15:17.615682 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:17.615647    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h" event={"ID":"88e52edd-9587-47c1-b9bc-e8d7a7c30460","Type":"ContainerStarted","Data":"2a76f11c2550dda402e634b883f0f2f0613f8625304dff28ce373b487a0ab237"}
Sep 15 19:15:17.615889 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:17.615691    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h" event={"ID":"88e52edd-9587-47c1-b9bc-e8d7a7c30460","Type":"ContainerStarted","Data":"40911e460ad49c3626a26c96d68962a9c17bad13f96acbbadf4234f60fc72425"}
Sep 15 19:15:17.617301 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:17.617273    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287" event={"ID":"e0d9a2f8-04b4-4728-b786-68974d219708","Type":"ContainerStarted","Data":"838afc67bca981f924e639b535a16f1037bfc3d27838893485564177588bb974"}
Sep 15 19:15:17.617441 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:17.617316    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287" event={"ID":"e0d9a2f8-04b4-4728-b786-68974d219708","Type":"ContainerStarted","Data":"e042aca5c3b7d86a98a2e24e21eae2721c06d17b00ab43162ab1ab1067c48a19"}
Sep 15 19:15:17.617441 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:17.617417    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"
Sep 15 19:15:17.634677 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:17.634630    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h" podStartSLOduration=1.634597439 podStartE2EDuration="1.634597439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:17.633157413 +0000 UTC m=+1459.120486379" watchObservedRunningTime="2026-09-15 19:15:17.634597439 +0000 UTC m=+1459.121926423"
Sep 15 19:15:17.650417 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:17.650363    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287" podStartSLOduration=2.321893624 podStartE2EDuration="2.65034962s" totalImagesPullingTime="328.455996ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:17.64909357 +0000 UTC m=+1459.136422564" watchObservedRunningTime="2026-09-15 19:15:17.65034962 +0000 UTC m=+1459.137678626"
Sep 15 19:15:23.628737 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:23.628709    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"
Sep 15 19:15:26.957992 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:26.957940    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:26.958563 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:26.958012    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:26.971030 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:26.971005    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:27.683110 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:27.683076    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:15:46.362793 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:46.362758    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"]
Sep 15 19:15:46.367384 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:46.367357    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"
Sep 15 19:15:46.369700 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:46.369679    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0a7ab7-serving-cert\""
Sep 15 19:15:46.374778 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:46.374756    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"]
Sep 15 19:15:46.512204 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:46.512149    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be5a39b-2af4-453b-8de6-06962e9cc44d-openshift-service-ca\") pod \"maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x\" (UID: \"5be5a39b-2af4-453b-8de6-06962e9cc44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"
Sep 15 19:15:46.512396 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:46.512232    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5be5a39b-2af4-453b-8de6-06962e9cc44d-metrics-certs\") pod \"maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x\" (UID: \"5be5a39b-2af4-453b-8de6-06962e9cc44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"
Sep 15 19:15:46.512396 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:46.512281    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5be5a39b-2af4-453b-8de6-06962e9cc44d-maas-api-tls\") pod \"maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x\" (UID: \"5be5a39b-2af4-453b-8de6-06962e9cc44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"
Sep 15 19:15:46.512396 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:46.512298    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llz9n\" (UniqueName: \"kubernetes.io/projected/5be5a39b-2af4-453b-8de6-06962e9cc44d-kube-api-access-llz9n\") pod \"maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x\" (UID: \"5be5a39b-2af4-453b-8de6-06962e9cc44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"
Sep 15 19:15:46.613561 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:46.613478    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5be5a39b-2af4-453b-8de6-06962e9cc44d-metrics-certs\") pod \"maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x\" (UID: \"5be5a39b-2af4-453b-8de6-06962e9cc44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"
Sep 15 19:15:46.613561 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:46.613531    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5be5a39b-2af4-453b-8de6-06962e9cc44d-maas-api-tls\") pod \"maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x\" (UID: \"5be5a39b-2af4-453b-8de6-06962e9cc44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"
Sep 15 19:15:46.613561 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:46.613548    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llz9n\" (UniqueName: \"kubernetes.io/projected/5be5a39b-2af4-453b-8de6-06962e9cc44d-kube-api-access-llz9n\") pod \"maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x\" (UID: \"5be5a39b-2af4-453b-8de6-06962e9cc44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"
Sep 15 19:15:46.613807 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:46.613629    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be5a39b-2af4-453b-8de6-06962e9cc44d-openshift-service-ca\") pod \"maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x\" (UID: \"5be5a39b-2af4-453b-8de6-06962e9cc44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"
Sep 15 19:15:46.614269 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:46.614251    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be5a39b-2af4-453b-8de6-06962e9cc44d-openshift-service-ca\") pod \"maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x\" (UID: \"5be5a39b-2af4-453b-8de6-06962e9cc44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"
Sep 15 19:15:46.616000 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:46.615978    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5be5a39b-2af4-453b-8de6-06962e9cc44d-maas-api-tls\") pod \"maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x\" (UID: \"5be5a39b-2af4-453b-8de6-06962e9cc44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"
Sep 15 19:15:46.616111 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:46.616030    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5be5a39b-2af4-453b-8de6-06962e9cc44d-metrics-certs\") pod \"maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x\" (UID: \"5be5a39b-2af4-453b-8de6-06962e9cc44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"
Sep 15 19:15:46.620893 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:46.620872    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llz9n\" (UniqueName: \"kubernetes.io/projected/5be5a39b-2af4-453b-8de6-06962e9cc44d-kube-api-access-llz9n\") pod \"maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x\" (UID: \"5be5a39b-2af4-453b-8de6-06962e9cc44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"
Sep 15 19:15:46.682031 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:46.681996    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"
Sep 15 19:15:46.960312 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:46.960217    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"]
Sep 15 19:15:46.963470 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:15:46.963439    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be5a39b_2af4_453b_8de6_06962e9cc44d.slice/crio-be90dfb94e194eea0bf40adf0cf0324c4276fd22c1fbb2d0941a07cc510dc27f WatchSource:0}: Error finding container be90dfb94e194eea0bf40adf0cf0324c4276fd22c1fbb2d0941a07cc510dc27f: Status 404 returned error can't find the container with id be90dfb94e194eea0bf40adf0cf0324c4276fd22c1fbb2d0941a07cc510dc27f
Sep 15 19:15:47.785701 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:47.785653    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x" event={"ID":"5be5a39b-2af4-453b-8de6-06962e9cc44d","Type":"ContainerStarted","Data":"19af084b2aca171790e823189732bb05b7cb01e44b2dd89e02c4cbbd1bfb4106"}
Sep 15 19:15:47.786094 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:47.785719    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x" event={"ID":"5be5a39b-2af4-453b-8de6-06962e9cc44d","Type":"ContainerStarted","Data":"be90dfb94e194eea0bf40adf0cf0324c4276fd22c1fbb2d0941a07cc510dc27f"}
Sep 15 19:15:47.786094 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:47.785747    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"
Sep 15 19:15:47.806685 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:47.806634    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x" podStartSLOduration=1.460776899 podStartE2EDuration="1.806600447s" totalImagesPullingTime="345.823548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:47.803754163 +0000 UTC m=+1489.291083148" watchObservedRunningTime="2026-09-15 19:15:47.806600447 +0000 UTC m=+1489.293929432"
Sep 15 19:15:53.796858 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:53.796831    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"
Sep 15 19:15:57.019673 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.019636    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"]
Sep 15 19:15:57.023825 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.023800    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.026997 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.026972    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-33ff04\"/\"sub-test-model-33ff04-kserve-self-signed-certs\""
Sep 15 19:15:57.027111 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.027007    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-33ff04\"/\"kube-root-ca.crt\""
Sep 15 19:15:57.027111 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.027022    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-33ff04\"/\"default-dockercfg-9prxg\""
Sep 15 19:15:57.027111 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.027022    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-33ff04\"/\"openshift-service-ca.crt\""
Sep 15 19:15:57.032407 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.032383    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"]
Sep 15 19:15:57.118470 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.118423    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-home\") pod \"sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.118655 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.118494    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade125f2-e25c-40ad-86ae-66ac1bb33467-tls-certs\") pod \"sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.118655 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.118531    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-dshm\") pod \"sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.118749 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.118680    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-tmp-dir\") pod \"sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.118749 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.118709    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65tsp\" (UniqueName: \"kubernetes.io/projected/ade125f2-e25c-40ad-86ae-66ac1bb33467-kube-api-access-65tsp\") pod \"sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.118826 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.118767    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-model-cache\") pod \"sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.220290 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.220256    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-model-cache\") pod \"sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.220481 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.220298    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-home\") pod \"sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.220481 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.220328    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade125f2-e25c-40ad-86ae-66ac1bb33467-tls-certs\") pod \"sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.220481 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.220356    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-dshm\") pod \"sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.220481 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.220462    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-tmp-dir\") pod \"sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.220746 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.220485    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65tsp\" (UniqueName: \"kubernetes.io/projected/ade125f2-e25c-40ad-86ae-66ac1bb33467-kube-api-access-65tsp\") pod \"sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.220746 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.220735    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-model-cache\") pod \"sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.220856 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.220769    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-home\") pod \"sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.220856 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.220804    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-tmp-dir\") pod \"sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.222713 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.222690    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-dshm\") pod \"sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.222987 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.222970    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade125f2-e25c-40ad-86ae-66ac1bb33467-tls-certs\") pod \"sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.227593 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.227569    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65tsp\" (UniqueName: \"kubernetes.io/projected/ade125f2-e25c-40ad-86ae-66ac1bb33467-kube-api-access-65tsp\") pod \"sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.334913 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.334834    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:15:57.607824 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.607796    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"]
Sep 15 19:15:57.609466 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:15:57.609439    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podade125f2_e25c_40ad_86ae_66ac1bb33467.slice/crio-0a330eaccf77c15b3e40538283b9db1e729a721893c92557a532e00a8c37d500 WatchSource:0}: Error finding container 0a330eaccf77c15b3e40538283b9db1e729a721893c92557a532e00a8c37d500: Status 404 returned error can't find the container with id 0a330eaccf77c15b3e40538283b9db1e729a721893c92557a532e00a8c37d500
Sep 15 19:15:57.837016 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.836978    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn" event={"ID":"ade125f2-e25c-40ad-86ae-66ac1bb33467","Type":"ContainerStarted","Data":"62a22c7ab5cf7fc23dd9b25e3b143a375887e477cf8acb88ff4c2e621c6321ff"}
Sep 15 19:15:57.837016 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.837022    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn" event={"ID":"ade125f2-e25c-40ad-86ae-66ac1bb33467","Type":"ContainerStarted","Data":"0a330eaccf77c15b3e40538283b9db1e729a721893c92557a532e00a8c37d500"}
Sep 15 19:15:57.852974 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:15:57.852929    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn" podStartSLOduration=0.852914673 podStartE2EDuration="852.914673ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:57.850975654 +0000 UTC m=+1499.338304639" watchObservedRunningTime="2026-09-15 19:15:57.852914673 +0000 UTC m=+1499.340243711"
Sep 15 19:16:07.335960 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:07.335930    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:16:07.336435 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:07.335979    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:16:07.348722 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:07.348701    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:16:07.899632 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:07.899592    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:16:30.746691 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:30.746654    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"]
Sep 15 19:16:30.747235 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:30.746974    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287" podUID="e0d9a2f8-04b4-4728-b786-68974d219708" containerName="maas-api" containerID="cri-o://838afc67bca981f924e639b535a16f1037bfc3d27838893485564177588bb974" gracePeriod=30
Sep 15 19:16:30.994370 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:30.994347    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"
Sep 15 19:16:31.015469 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.015376    2555 generic.go:356] "Generic (PLEG): container finished" podID="e0d9a2f8-04b4-4728-b786-68974d219708" containerID="838afc67bca981f924e639b535a16f1037bfc3d27838893485564177588bb974" exitCode=0
Sep 15 19:16:31.015666 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.015482    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287" event={"ID":"e0d9a2f8-04b4-4728-b786-68974d219708","Type":"ContainerDied","Data":"838afc67bca981f924e639b535a16f1037bfc3d27838893485564177588bb974"}
Sep 15 19:16:31.015666 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.015488    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"
Sep 15 19:16:31.015666 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.015531    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287" event={"ID":"e0d9a2f8-04b4-4728-b786-68974d219708","Type":"ContainerDied","Data":"e042aca5c3b7d86a98a2e24e21eae2721c06d17b00ab43162ab1ab1067c48a19"}
Sep 15 19:16:31.015666 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.015563    2555 scope.go:117] "RemoveContainer" containerID="838afc67bca981f924e639b535a16f1037bfc3d27838893485564177588bb974"
Sep 15 19:16:31.025114 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.025095    2555 scope.go:117] "RemoveContainer" containerID="838afc67bca981f924e639b535a16f1037bfc3d27838893485564177588bb974"
Sep 15 19:16:31.025414 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:16:31.025389    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"838afc67bca981f924e639b535a16f1037bfc3d27838893485564177588bb974\": container with ID starting with 838afc67bca981f924e639b535a16f1037bfc3d27838893485564177588bb974 not found: ID does not exist" containerID="838afc67bca981f924e639b535a16f1037bfc3d27838893485564177588bb974"
Sep 15 19:16:31.025471 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.025425    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838afc67bca981f924e639b535a16f1037bfc3d27838893485564177588bb974"} err="failed to get container status \"838afc67bca981f924e639b535a16f1037bfc3d27838893485564177588bb974\": rpc error: code = NotFound desc = could not find container \"838afc67bca981f924e639b535a16f1037bfc3d27838893485564177588bb974\": container with ID starting with 838afc67bca981f924e639b535a16f1037bfc3d27838893485564177588bb974 not found: ID does not exist"
Sep 15 19:16:31.065478 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.065437    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0d9a2f8-04b4-4728-b786-68974d219708-maas-api-tls\") pod \"e0d9a2f8-04b4-4728-b786-68974d219708\" (UID: \"e0d9a2f8-04b4-4728-b786-68974d219708\") "
Sep 15 19:16:31.065743 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.065502    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fbvd\" (UniqueName: \"kubernetes.io/projected/e0d9a2f8-04b4-4728-b786-68974d219708-kube-api-access-6fbvd\") pod \"e0d9a2f8-04b4-4728-b786-68974d219708\" (UID: \"e0d9a2f8-04b4-4728-b786-68974d219708\") "
Sep 15 19:16:31.065743 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.065569    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d9a2f8-04b4-4728-b786-68974d219708-openshift-service-ca\") pod \"e0d9a2f8-04b4-4728-b786-68974d219708\" (UID: \"e0d9a2f8-04b4-4728-b786-68974d219708\") "
Sep 15 19:16:31.065743 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.065593    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0d9a2f8-04b4-4728-b786-68974d219708-metrics-certs\") pod \"e0d9a2f8-04b4-4728-b786-68974d219708\" (UID: \"e0d9a2f8-04b4-4728-b786-68974d219708\") "
Sep 15 19:16:31.065938 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.065910    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d9a2f8-04b4-4728-b786-68974d219708-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e0d9a2f8-04b4-4728-b786-68974d219708" (UID: "e0d9a2f8-04b4-4728-b786-68974d219708"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:16:31.067574 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.067545    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d9a2f8-04b4-4728-b786-68974d219708-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e0d9a2f8-04b4-4728-b786-68974d219708" (UID: "e0d9a2f8-04b4-4728-b786-68974d219708"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:16:31.067733 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.067712    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d9a2f8-04b4-4728-b786-68974d219708-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e0d9a2f8-04b4-4728-b786-68974d219708" (UID: "e0d9a2f8-04b4-4728-b786-68974d219708"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:16:31.067810 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.067772    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d9a2f8-04b4-4728-b786-68974d219708-kube-api-access-6fbvd" (OuterVolumeSpecName: "kube-api-access-6fbvd") pod "e0d9a2f8-04b4-4728-b786-68974d219708" (UID: "e0d9a2f8-04b4-4728-b786-68974d219708"). InnerVolumeSpecName "kube-api-access-6fbvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:16:31.167195 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.167156    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0d9a2f8-04b4-4728-b786-68974d219708-maas-api-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:16:31.167195 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.167195    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fbvd\" (UniqueName: \"kubernetes.io/projected/e0d9a2f8-04b4-4728-b786-68974d219708-kube-api-access-6fbvd\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:16:31.167407 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.167210    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d9a2f8-04b4-4728-b786-68974d219708-openshift-service-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:16:31.167407 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.167226    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0d9a2f8-04b4-4728-b786-68974d219708-metrics-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:16:31.337167 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.337132    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"]
Sep 15 19:16:31.343631 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:31.343586    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-910e11-69795bd8f5-hw287"]
Sep 15 19:16:33.040936 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:33.040899    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0d9a2f8-04b4-4728-b786-68974d219708" path="/var/lib/kubelet/pods/e0d9a2f8-04b4-4728-b786-68974d219708/volumes"
Sep 15 19:16:45.524060 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:45.524027    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"]
Sep 15 19:16:46.755749 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:46.755712    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"]
Sep 15 19:16:53.139785 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:53.139750    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"]
Sep 15 19:16:53.140342 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:53.139967    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x" podUID="5be5a39b-2af4-453b-8de6-06962e9cc44d" containerName="maas-api" containerID="cri-o://19af084b2aca171790e823189732bb05b7cb01e44b2dd89e02c4cbbd1bfb4106" gracePeriod=30
Sep 15 19:16:53.402518 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:53.402447    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"
Sep 15 19:16:53.488690 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:53.488658    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llz9n\" (UniqueName: \"kubernetes.io/projected/5be5a39b-2af4-453b-8de6-06962e9cc44d-kube-api-access-llz9n\") pod \"5be5a39b-2af4-453b-8de6-06962e9cc44d\" (UID: \"5be5a39b-2af4-453b-8de6-06962e9cc44d\") "
Sep 15 19:16:53.488883 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:53.488792    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5be5a39b-2af4-453b-8de6-06962e9cc44d-maas-api-tls\") pod \"5be5a39b-2af4-453b-8de6-06962e9cc44d\" (UID: \"5be5a39b-2af4-453b-8de6-06962e9cc44d\") "
Sep 15 19:16:53.488883 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:53.488832    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5be5a39b-2af4-453b-8de6-06962e9cc44d-metrics-certs\") pod \"5be5a39b-2af4-453b-8de6-06962e9cc44d\" (UID: \"5be5a39b-2af4-453b-8de6-06962e9cc44d\") "
Sep 15 19:16:53.488883 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:53.488860    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be5a39b-2af4-453b-8de6-06962e9cc44d-openshift-service-ca\") pod \"5be5a39b-2af4-453b-8de6-06962e9cc44d\" (UID: \"5be5a39b-2af4-453b-8de6-06962e9cc44d\") "
Sep 15 19:16:53.489358 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:53.489326    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be5a39b-2af4-453b-8de6-06962e9cc44d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5be5a39b-2af4-453b-8de6-06962e9cc44d" (UID: "5be5a39b-2af4-453b-8de6-06962e9cc44d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:16:53.491051 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:53.490994    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5be5a39b-2af4-453b-8de6-06962e9cc44d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5be5a39b-2af4-453b-8de6-06962e9cc44d" (UID: "5be5a39b-2af4-453b-8de6-06962e9cc44d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:16:53.491171 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:53.491049    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5be5a39b-2af4-453b-8de6-06962e9cc44d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5be5a39b-2af4-453b-8de6-06962e9cc44d" (UID: "5be5a39b-2af4-453b-8de6-06962e9cc44d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:16:53.491171 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:53.491116    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be5a39b-2af4-453b-8de6-06962e9cc44d-kube-api-access-llz9n" (OuterVolumeSpecName: "kube-api-access-llz9n") pod "5be5a39b-2af4-453b-8de6-06962e9cc44d" (UID: "5be5a39b-2af4-453b-8de6-06962e9cc44d"). InnerVolumeSpecName "kube-api-access-llz9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:16:53.589735 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:53.589696    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5be5a39b-2af4-453b-8de6-06962e9cc44d-maas-api-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:16:53.589735 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:53.589728    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5be5a39b-2af4-453b-8de6-06962e9cc44d-metrics-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:16:53.589735 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:53.589738    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be5a39b-2af4-453b-8de6-06962e9cc44d-openshift-service-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:16:53.589995 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:53.589750    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llz9n\" (UniqueName: \"kubernetes.io/projected/5be5a39b-2af4-453b-8de6-06962e9cc44d-kube-api-access-llz9n\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:16:54.139751 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:54.139717    2555 generic.go:356] "Generic (PLEG): container finished" podID="5be5a39b-2af4-453b-8de6-06962e9cc44d" containerID="19af084b2aca171790e823189732bb05b7cb01e44b2dd89e02c4cbbd1bfb4106" exitCode=0
Sep 15 19:16:54.139942 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:54.139769    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"
Sep 15 19:16:54.139942 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:54.139772    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x" event={"ID":"5be5a39b-2af4-453b-8de6-06962e9cc44d","Type":"ContainerDied","Data":"19af084b2aca171790e823189732bb05b7cb01e44b2dd89e02c4cbbd1bfb4106"}
Sep 15 19:16:54.139942 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:54.139907    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x" event={"ID":"5be5a39b-2af4-453b-8de6-06962e9cc44d","Type":"ContainerDied","Data":"be90dfb94e194eea0bf40adf0cf0324c4276fd22c1fbb2d0941a07cc510dc27f"}
Sep 15 19:16:54.140406 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:54.139948    2555 scope.go:117] "RemoveContainer" containerID="19af084b2aca171790e823189732bb05b7cb01e44b2dd89e02c4cbbd1bfb4106"
Sep 15 19:16:54.149564 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:54.149543    2555 scope.go:117] "RemoveContainer" containerID="19af084b2aca171790e823189732bb05b7cb01e44b2dd89e02c4cbbd1bfb4106"
Sep 15 19:16:54.149829 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:16:54.149812    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19af084b2aca171790e823189732bb05b7cb01e44b2dd89e02c4cbbd1bfb4106\": container with ID starting with 19af084b2aca171790e823189732bb05b7cb01e44b2dd89e02c4cbbd1bfb4106 not found: ID does not exist" containerID="19af084b2aca171790e823189732bb05b7cb01e44b2dd89e02c4cbbd1bfb4106"
Sep 15 19:16:54.149912 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:54.149835    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19af084b2aca171790e823189732bb05b7cb01e44b2dd89e02c4cbbd1bfb4106"} err="failed to get container status \"19af084b2aca171790e823189732bb05b7cb01e44b2dd89e02c4cbbd1bfb4106\": rpc error: code = NotFound desc = could not find container \"19af084b2aca171790e823189732bb05b7cb01e44b2dd89e02c4cbbd1bfb4106\": container with ID starting with 19af084b2aca171790e823189732bb05b7cb01e44b2dd89e02c4cbbd1bfb4106 not found: ID does not exist"
Sep 15 19:16:54.160833 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:54.160802    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"]
Sep 15 19:16:54.162734 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:54.162719    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0a7ab7-89c7c49c5-xhs5x"]
Sep 15 19:16:55.040662 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:16:55.040632    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5be5a39b-2af4-453b-8de6-06962e9cc44d" path="/var/lib/kubelet/pods/5be5a39b-2af4-453b-8de6-06962e9cc44d/volumes"
Sep 15 19:17:00.543877 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:00.543837    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h" podUID="88e52edd-9587-47c1-b9bc-e8d7a7c30460" containerName="main" containerID="cri-o://2a76f11c2550dda402e634b883f0f2f0613f8625304dff28ce373b487a0ab237" gracePeriod=45
Sep 15 19:17:00.800468 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:00.800405    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:17:00.963182 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:00.963144    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e52edd-9587-47c1-b9bc-e8d7a7c30460-tls-certs\") pod \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") "
Sep 15 19:17:00.963182 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:00.963189    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-tmp-dir\") pod \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") "
Sep 15 19:17:00.963437 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:00.963221    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-home\") pod \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") "
Sep 15 19:17:00.963437 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:00.963257    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-dshm\") pod \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") "
Sep 15 19:17:00.963437 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:00.963291    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2gkc\" (UniqueName: \"kubernetes.io/projected/88e52edd-9587-47c1-b9bc-e8d7a7c30460-kube-api-access-f2gkc\") pod \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") "
Sep 15 19:17:00.963437 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:00.963351    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-model-cache\") pod \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\" (UID: \"88e52edd-9587-47c1-b9bc-e8d7a7c30460\") "
Sep 15 19:17:00.963683 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:00.963506    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-home" (OuterVolumeSpecName: "home") pod "88e52edd-9587-47c1-b9bc-e8d7a7c30460" (UID: "88e52edd-9587-47c1-b9bc-e8d7a7c30460"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:00.963683 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:00.963522    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88e52edd-9587-47c1-b9bc-e8d7a7c30460" (UID: "88e52edd-9587-47c1-b9bc-e8d7a7c30460"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:00.963834 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:00.963714    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-model-cache" (OuterVolumeSpecName: "model-cache") pod "88e52edd-9587-47c1-b9bc-e8d7a7c30460" (UID: "88e52edd-9587-47c1-b9bc-e8d7a7c30460"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:00.963834 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:00.963747    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-tmp-dir\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:00.963834 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:00.963762    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-home\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:00.965544 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:00.965518    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e52edd-9587-47c1-b9bc-e8d7a7c30460-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88e52edd-9587-47c1-b9bc-e8d7a7c30460" (UID: "88e52edd-9587-47c1-b9bc-e8d7a7c30460"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:17:00.965544 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:00.965527    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-dshm" (OuterVolumeSpecName: "dshm") pod "88e52edd-9587-47c1-b9bc-e8d7a7c30460" (UID: "88e52edd-9587-47c1-b9bc-e8d7a7c30460"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:00.965756 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:00.965560    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e52edd-9587-47c1-b9bc-e8d7a7c30460-kube-api-access-f2gkc" (OuterVolumeSpecName: "kube-api-access-f2gkc") pod "88e52edd-9587-47c1-b9bc-e8d7a7c30460" (UID: "88e52edd-9587-47c1-b9bc-e8d7a7c30460"). InnerVolumeSpecName "kube-api-access-f2gkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:17:01.064431 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:01.064362    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2gkc\" (UniqueName: \"kubernetes.io/projected/88e52edd-9587-47c1-b9bc-e8d7a7c30460-kube-api-access-f2gkc\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:01.064431 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:01.064389    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-model-cache\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:01.064431 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:01.064399    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e52edd-9587-47c1-b9bc-e8d7a7c30460-tls-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:01.064431 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:01.064408    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e52edd-9587-47c1-b9bc-e8d7a7c30460-dshm\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:01.180554 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:01.180520    2555 generic.go:356] "Generic (PLEG): container finished" podID="88e52edd-9587-47c1-b9bc-e8d7a7c30460" containerID="2a76f11c2550dda402e634b883f0f2f0613f8625304dff28ce373b487a0ab237" exitCode=0
Sep 15 19:17:01.180724 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:01.180567    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h" event={"ID":"88e52edd-9587-47c1-b9bc-e8d7a7c30460","Type":"ContainerDied","Data":"2a76f11c2550dda402e634b883f0f2f0613f8625304dff28ce373b487a0ab237"}
Sep 15 19:17:01.180724 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:01.180601    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h" event={"ID":"88e52edd-9587-47c1-b9bc-e8d7a7c30460","Type":"ContainerDied","Data":"40911e460ad49c3626a26c96d68962a9c17bad13f96acbbadf4234f60fc72425"}
Sep 15 19:17:01.180724 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:01.180571    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"
Sep 15 19:17:01.180724 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:01.180652    2555 scope.go:117] "RemoveContainer" containerID="2a76f11c2550dda402e634b883f0f2f0613f8625304dff28ce373b487a0ab237"
Sep 15 19:17:01.191981 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:01.191959    2555 scope.go:117] "RemoveContainer" containerID="2a76f11c2550dda402e634b883f0f2f0613f8625304dff28ce373b487a0ab237"
Sep 15 19:17:01.192290 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:17:01.192267    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a76f11c2550dda402e634b883f0f2f0613f8625304dff28ce373b487a0ab237\": container with ID starting with 2a76f11c2550dda402e634b883f0f2f0613f8625304dff28ce373b487a0ab237 not found: ID does not exist" containerID="2a76f11c2550dda402e634b883f0f2f0613f8625304dff28ce373b487a0ab237"
Sep 15 19:17:01.192373 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:01.192300    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a76f11c2550dda402e634b883f0f2f0613f8625304dff28ce373b487a0ab237"} err="failed to get container status \"2a76f11c2550dda402e634b883f0f2f0613f8625304dff28ce373b487a0ab237\": rpc error: code = NotFound desc = could not find container \"2a76f11c2550dda402e634b883f0f2f0613f8625304dff28ce373b487a0ab237\": container with ID starting with 2a76f11c2550dda402e634b883f0f2f0613f8625304dff28ce373b487a0ab237 not found: ID does not exist"
Sep 15 19:17:01.196213 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:01.196190    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"]
Sep 15 19:17:01.199675 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:01.199654    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-7073b8/sub-test-model-7073b8-kserve-dcbff5f4f-j8b9h"]
Sep 15 19:17:01.776400 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:01.776363    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn" podUID="ade125f2-e25c-40ad-86ae-66ac1bb33467" containerName="main" containerID="cri-o://62a22c7ab5cf7fc23dd9b25e3b143a375887e477cf8acb88ff4c2e621c6321ff" gracePeriod=45
Sep 15 19:17:02.024877 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.024853    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:17:02.075654 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.075564    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-model-cache\") pod \"ade125f2-e25c-40ad-86ae-66ac1bb33467\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") "
Sep 15 19:17:02.075654 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.075604    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-tmp-dir\") pod \"ade125f2-e25c-40ad-86ae-66ac1bb33467\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") "
Sep 15 19:17:02.075867 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.075671    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-home\") pod \"ade125f2-e25c-40ad-86ae-66ac1bb33467\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") "
Sep 15 19:17:02.075867 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.075716    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-dshm\") pod \"ade125f2-e25c-40ad-86ae-66ac1bb33467\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") "
Sep 15 19:17:02.075867 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.075771    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65tsp\" (UniqueName: \"kubernetes.io/projected/ade125f2-e25c-40ad-86ae-66ac1bb33467-kube-api-access-65tsp\") pod \"ade125f2-e25c-40ad-86ae-66ac1bb33467\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") "
Sep 15 19:17:02.075867 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.075790    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade125f2-e25c-40ad-86ae-66ac1bb33467-tls-certs\") pod \"ade125f2-e25c-40ad-86ae-66ac1bb33467\" (UID: \"ade125f2-e25c-40ad-86ae-66ac1bb33467\") "
Sep 15 19:17:02.075867 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.075809    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ade125f2-e25c-40ad-86ae-66ac1bb33467" (UID: "ade125f2-e25c-40ad-86ae-66ac1bb33467"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:02.075867 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.075848    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-model-cache" (OuterVolumeSpecName: "model-cache") pod "ade125f2-e25c-40ad-86ae-66ac1bb33467" (UID: "ade125f2-e25c-40ad-86ae-66ac1bb33467"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:02.076119 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.075946    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-home" (OuterVolumeSpecName: "home") pod "ade125f2-e25c-40ad-86ae-66ac1bb33467" (UID: "ade125f2-e25c-40ad-86ae-66ac1bb33467"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:02.076919 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.076848    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-home\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:02.076919 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.076874    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-model-cache\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:02.076919 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.076896    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-tmp-dir\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:02.080278 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.080247    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-dshm" (OuterVolumeSpecName: "dshm") pod "ade125f2-e25c-40ad-86ae-66ac1bb33467" (UID: "ade125f2-e25c-40ad-86ae-66ac1bb33467"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:02.087164 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.086495    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade125f2-e25c-40ad-86ae-66ac1bb33467-kube-api-access-65tsp" (OuterVolumeSpecName: "kube-api-access-65tsp") pod "ade125f2-e25c-40ad-86ae-66ac1bb33467" (UID: "ade125f2-e25c-40ad-86ae-66ac1bb33467"). InnerVolumeSpecName "kube-api-access-65tsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:17:02.087843 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.087821    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade125f2-e25c-40ad-86ae-66ac1bb33467-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ade125f2-e25c-40ad-86ae-66ac1bb33467" (UID: "ade125f2-e25c-40ad-86ae-66ac1bb33467"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:17:02.177826 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.177793    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade125f2-e25c-40ad-86ae-66ac1bb33467-dshm\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:02.178010 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.177844    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65tsp\" (UniqueName: \"kubernetes.io/projected/ade125f2-e25c-40ad-86ae-66ac1bb33467-kube-api-access-65tsp\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:02.178010 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.177856    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade125f2-e25c-40ad-86ae-66ac1bb33467-tls-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:02.187539 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.187509    2555 generic.go:356] "Generic (PLEG): container finished" podID="ade125f2-e25c-40ad-86ae-66ac1bb33467" containerID="62a22c7ab5cf7fc23dd9b25e3b143a375887e477cf8acb88ff4c2e621c6321ff" exitCode=0
Sep 15 19:17:02.187703 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.187563    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"
Sep 15 19:17:02.187703 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.187597    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn" event={"ID":"ade125f2-e25c-40ad-86ae-66ac1bb33467","Type":"ContainerDied","Data":"62a22c7ab5cf7fc23dd9b25e3b143a375887e477cf8acb88ff4c2e621c6321ff"}
Sep 15 19:17:02.187703 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.187660    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn" event={"ID":"ade125f2-e25c-40ad-86ae-66ac1bb33467","Type":"ContainerDied","Data":"0a330eaccf77c15b3e40538283b9db1e729a721893c92557a532e00a8c37d500"}
Sep 15 19:17:02.187703 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.187678    2555 scope.go:117] "RemoveContainer" containerID="62a22c7ab5cf7fc23dd9b25e3b143a375887e477cf8acb88ff4c2e621c6321ff"
Sep 15 19:17:02.198116 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.198097    2555 scope.go:117] "RemoveContainer" containerID="62a22c7ab5cf7fc23dd9b25e3b143a375887e477cf8acb88ff4c2e621c6321ff"
Sep 15 19:17:02.198404 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:17:02.198384    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a22c7ab5cf7fc23dd9b25e3b143a375887e477cf8acb88ff4c2e621c6321ff\": container with ID starting with 62a22c7ab5cf7fc23dd9b25e3b143a375887e477cf8acb88ff4c2e621c6321ff not found: ID does not exist" containerID="62a22c7ab5cf7fc23dd9b25e3b143a375887e477cf8acb88ff4c2e621c6321ff"
Sep 15 19:17:02.198468 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.198412    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a22c7ab5cf7fc23dd9b25e3b143a375887e477cf8acb88ff4c2e621c6321ff"} err="failed to get container status \"62a22c7ab5cf7fc23dd9b25e3b143a375887e477cf8acb88ff4c2e621c6321ff\": rpc error: code = NotFound desc = could not find container \"62a22c7ab5cf7fc23dd9b25e3b143a375887e477cf8acb88ff4c2e621c6321ff\": container with ID starting with 62a22c7ab5cf7fc23dd9b25e3b143a375887e477cf8acb88ff4c2e621c6321ff not found: ID does not exist"
Sep 15 19:17:02.208796 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.208763    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"]
Sep 15 19:17:02.212099 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:02.212076    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-33ff04/sub-test-model-33ff04-kserve-6ff6d485f5-pn8mn"]
Sep 15 19:17:03.040134 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:03.040103    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88e52edd-9587-47c1-b9bc-e8d7a7c30460" path="/var/lib/kubelet/pods/88e52edd-9587-47c1-b9bc-e8d7a7c30460/volumes"
Sep 15 19:17:03.040517 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:03.040505    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ade125f2-e25c-40ad-86ae-66ac1bb33467" path="/var/lib/kubelet/pods/ade125f2-e25c-40ad-86ae-66ac1bb33467/volumes"
Sep 15 19:17:10.592179 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:10.592139    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"]
Sep 15 19:17:10.592721 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:10.592403    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz" podUID="dd3216a3-8c92-4e1b-884a-bf5e90d1f772" containerName="maas-api" containerID="cri-o://7fb9c409ee1ea066d38a0157bb56769c68ce9ec81b0dc72213ef029305c207d1" gracePeriod=30
Sep 15 19:17:10.840423 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:10.840400    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"
Sep 15 19:17:10.960507 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:10.960413    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-maas-api-tls\") pod \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\" (UID: \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\") "
Sep 15 19:17:10.960507 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:10.960449    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qkq2\" (UniqueName: \"kubernetes.io/projected/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-kube-api-access-9qkq2\") pod \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\" (UID: \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\") "
Sep 15 19:17:10.960773 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:10.960591    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-openshift-service-ca\") pod \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\" (UID: \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\") "
Sep 15 19:17:10.960773 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:10.960734    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-metrics-certs\") pod \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\" (UID: \"dd3216a3-8c92-4e1b-884a-bf5e90d1f772\") "
Sep 15 19:17:10.960982 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:10.960950    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dd3216a3-8c92-4e1b-884a-bf5e90d1f772" (UID: "dd3216a3-8c92-4e1b-884a-bf5e90d1f772"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:17:10.961122 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:10.961102    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-openshift-service-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:10.963036 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:10.963003    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-kube-api-access-9qkq2" (OuterVolumeSpecName: "kube-api-access-9qkq2") pod "dd3216a3-8c92-4e1b-884a-bf5e90d1f772" (UID: "dd3216a3-8c92-4e1b-884a-bf5e90d1f772"). InnerVolumeSpecName "kube-api-access-9qkq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:17:10.963408 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:10.963386    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dd3216a3-8c92-4e1b-884a-bf5e90d1f772" (UID: "dd3216a3-8c92-4e1b-884a-bf5e90d1f772"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:17:10.963513 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:10.963419    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dd3216a3-8c92-4e1b-884a-bf5e90d1f772" (UID: "dd3216a3-8c92-4e1b-884a-bf5e90d1f772"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:17:11.062213 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:11.062172    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-metrics-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:11.062213 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:11.062199    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-maas-api-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:11.062213 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:11.062208    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qkq2\" (UniqueName: \"kubernetes.io/projected/dd3216a3-8c92-4e1b-884a-bf5e90d1f772-kube-api-access-9qkq2\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:11.237418 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:11.237327    2555 generic.go:356] "Generic (PLEG): container finished" podID="dd3216a3-8c92-4e1b-884a-bf5e90d1f772" containerID="7fb9c409ee1ea066d38a0157bb56769c68ce9ec81b0dc72213ef029305c207d1" exitCode=0
Sep 15 19:17:11.237418 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:11.237378    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"
Sep 15 19:17:11.237418 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:11.237404    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz" event={"ID":"dd3216a3-8c92-4e1b-884a-bf5e90d1f772","Type":"ContainerDied","Data":"7fb9c409ee1ea066d38a0157bb56769c68ce9ec81b0dc72213ef029305c207d1"}
Sep 15 19:17:11.237737 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:11.237458    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz" event={"ID":"dd3216a3-8c92-4e1b-884a-bf5e90d1f772","Type":"ContainerDied","Data":"2c47a2bbe1525acadc088acd68b56a8e1c9e5fe82e14c6d36da1f016068a422e"}
Sep 15 19:17:11.237737 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:11.237479    2555 scope.go:117] "RemoveContainer" containerID="7fb9c409ee1ea066d38a0157bb56769c68ce9ec81b0dc72213ef029305c207d1"
Sep 15 19:17:11.246103 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:11.246088    2555 scope.go:117] "RemoveContainer" containerID="7fb9c409ee1ea066d38a0157bb56769c68ce9ec81b0dc72213ef029305c207d1"
Sep 15 19:17:11.246389 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:17:11.246373    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb9c409ee1ea066d38a0157bb56769c68ce9ec81b0dc72213ef029305c207d1\": container with ID starting with 7fb9c409ee1ea066d38a0157bb56769c68ce9ec81b0dc72213ef029305c207d1 not found: ID does not exist" containerID="7fb9c409ee1ea066d38a0157bb56769c68ce9ec81b0dc72213ef029305c207d1"
Sep 15 19:17:11.246441 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:11.246397    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb9c409ee1ea066d38a0157bb56769c68ce9ec81b0dc72213ef029305c207d1"} err="failed to get container status \"7fb9c409ee1ea066d38a0157bb56769c68ce9ec81b0dc72213ef029305c207d1\": rpc error: code = NotFound desc = could not find container \"7fb9c409ee1ea066d38a0157bb56769c68ce9ec81b0dc72213ef029305c207d1\": container with ID starting with 7fb9c409ee1ea066d38a0157bb56769c68ce9ec81b0dc72213ef029305c207d1 not found: ID does not exist"
Sep 15 19:17:11.274189 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:11.274154    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"]
Sep 15 19:17:11.290126 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:11.290103    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4d299f-765b4d8756-vwxmz"]
Sep 15 19:17:13.041024 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:13.040991    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd3216a3-8c92-4e1b-884a-bf5e90d1f772" path="/var/lib/kubelet/pods/dd3216a3-8c92-4e1b-884a-bf5e90d1f772/volumes"
Sep 15 19:17:28.034020 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.033970    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"]
Sep 15 19:17:28.035109 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.034244    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s" podUID="d08b3386-95ee-42ab-8fd6-5787bed9336a" containerName="maas-api" containerID="cri-o://6b197ace1acd653f51e362fafc7ac3657f474cefcb138434177076f576b71887" gracePeriod=30
Sep 15 19:17:28.286595 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.286533    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"
Sep 15 19:17:28.328867 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.328835    2555 generic.go:356] "Generic (PLEG): container finished" podID="d08b3386-95ee-42ab-8fd6-5787bed9336a" containerID="6b197ace1acd653f51e362fafc7ac3657f474cefcb138434177076f576b71887" exitCode=0
Sep 15 19:17:28.329043 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.328882    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"
Sep 15 19:17:28.329043 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.328922    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s" event={"ID":"d08b3386-95ee-42ab-8fd6-5787bed9336a","Type":"ContainerDied","Data":"6b197ace1acd653f51e362fafc7ac3657f474cefcb138434177076f576b71887"}
Sep 15 19:17:28.329043 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.328979    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s" event={"ID":"d08b3386-95ee-42ab-8fd6-5787bed9336a","Type":"ContainerDied","Data":"6c0dca8f638f8a7f600962e130d48542a4cea6779ad0bd542c1331d4b4c3b832"}
Sep 15 19:17:28.329043 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.329004    2555 scope.go:117] "RemoveContainer" containerID="6b197ace1acd653f51e362fafc7ac3657f474cefcb138434177076f576b71887"
Sep 15 19:17:28.338136 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.338121    2555 scope.go:117] "RemoveContainer" containerID="6b197ace1acd653f51e362fafc7ac3657f474cefcb138434177076f576b71887"
Sep 15 19:17:28.338424 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:17:28.338404    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b197ace1acd653f51e362fafc7ac3657f474cefcb138434177076f576b71887\": container with ID starting with 6b197ace1acd653f51e362fafc7ac3657f474cefcb138434177076f576b71887 not found: ID does not exist" containerID="6b197ace1acd653f51e362fafc7ac3657f474cefcb138434177076f576b71887"
Sep 15 19:17:28.338509 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.338432    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b197ace1acd653f51e362fafc7ac3657f474cefcb138434177076f576b71887"} err="failed to get container status \"6b197ace1acd653f51e362fafc7ac3657f474cefcb138434177076f576b71887\": rpc error: code = NotFound desc = could not find container \"6b197ace1acd653f51e362fafc7ac3657f474cefcb138434177076f576b71887\": container with ID starting with 6b197ace1acd653f51e362fafc7ac3657f474cefcb138434177076f576b71887 not found: ID does not exist"
Sep 15 19:17:28.438435 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.438399    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08b3386-95ee-42ab-8fd6-5787bed9336a-maas-api-tls\") pod \"d08b3386-95ee-42ab-8fd6-5787bed9336a\" (UID: \"d08b3386-95ee-42ab-8fd6-5787bed9336a\") "
Sep 15 19:17:28.438435 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.438445    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08b3386-95ee-42ab-8fd6-5787bed9336a-metrics-certs\") pod \"d08b3386-95ee-42ab-8fd6-5787bed9336a\" (UID: \"d08b3386-95ee-42ab-8fd6-5787bed9336a\") "
Sep 15 19:17:28.438728 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.438488    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08b3386-95ee-42ab-8fd6-5787bed9336a-openshift-service-ca\") pod \"d08b3386-95ee-42ab-8fd6-5787bed9336a\" (UID: \"d08b3386-95ee-42ab-8fd6-5787bed9336a\") "
Sep 15 19:17:28.438728 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.438570    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27kq8\" (UniqueName: \"kubernetes.io/projected/d08b3386-95ee-42ab-8fd6-5787bed9336a-kube-api-access-27kq8\") pod \"d08b3386-95ee-42ab-8fd6-5787bed9336a\" (UID: \"d08b3386-95ee-42ab-8fd6-5787bed9336a\") "
Sep 15 19:17:28.439004 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.438975    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08b3386-95ee-42ab-8fd6-5787bed9336a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d08b3386-95ee-42ab-8fd6-5787bed9336a" (UID: "d08b3386-95ee-42ab-8fd6-5787bed9336a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:17:28.440683 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.440649    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08b3386-95ee-42ab-8fd6-5787bed9336a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d08b3386-95ee-42ab-8fd6-5787bed9336a" (UID: "d08b3386-95ee-42ab-8fd6-5787bed9336a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:17:28.440796 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.440738    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08b3386-95ee-42ab-8fd6-5787bed9336a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d08b3386-95ee-42ab-8fd6-5787bed9336a" (UID: "d08b3386-95ee-42ab-8fd6-5787bed9336a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:17:28.440847 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.440801    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08b3386-95ee-42ab-8fd6-5787bed9336a-kube-api-access-27kq8" (OuterVolumeSpecName: "kube-api-access-27kq8") pod "d08b3386-95ee-42ab-8fd6-5787bed9336a" (UID: "d08b3386-95ee-42ab-8fd6-5787bed9336a"). InnerVolumeSpecName "kube-api-access-27kq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:17:28.539498 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.539409    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08b3386-95ee-42ab-8fd6-5787bed9336a-metrics-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:28.539498 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.539439    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08b3386-95ee-42ab-8fd6-5787bed9336a-openshift-service-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:28.539498 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.539450    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27kq8\" (UniqueName: \"kubernetes.io/projected/d08b3386-95ee-42ab-8fd6-5787bed9336a-kube-api-access-27kq8\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:28.539498 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.539458    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08b3386-95ee-42ab-8fd6-5787bed9336a-maas-api-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:17:28.650745 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.650717    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"]
Sep 15 19:17:28.656228 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:28.656202    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-501bed-6558976c6d-2ws6s"]
Sep 15 19:17:29.039910 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:29.039876    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d08b3386-95ee-42ab-8fd6-5787bed9336a" path="/var/lib/kubelet/pods/d08b3386-95ee-42ab-8fd6-5787bed9336a/volumes"
Sep 15 19:17:35.142723 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.142686    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"]
Sep 15 19:17:35.143174 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.143158    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d08b3386-95ee-42ab-8fd6-5787bed9336a" containerName="maas-api"
Sep 15 19:17:35.143225 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.143176    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d08b3386-95ee-42ab-8fd6-5787bed9336a" containerName="maas-api"
Sep 15 19:17:35.143225 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.143188    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade125f2-e25c-40ad-86ae-66ac1bb33467" containerName="main"
Sep 15 19:17:35.143225 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.143194    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade125f2-e25c-40ad-86ae-66ac1bb33467" containerName="main"
Sep 15 19:17:35.143225 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.143209    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd3216a3-8c92-4e1b-884a-bf5e90d1f772" containerName="maas-api"
Sep 15 19:17:35.143225 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.143218    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd3216a3-8c92-4e1b-884a-bf5e90d1f772" containerName="maas-api"
Sep 15 19:17:35.143382 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.143228    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5be5a39b-2af4-453b-8de6-06962e9cc44d" containerName="maas-api"
Sep 15 19:17:35.143382 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.143233    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5be5a39b-2af4-453b-8de6-06962e9cc44d" containerName="maas-api"
Sep 15 19:17:35.143382 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.143254    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e52edd-9587-47c1-b9bc-e8d7a7c30460" containerName="main"
Sep 15 19:17:35.143382 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.143260    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e52edd-9587-47c1-b9bc-e8d7a7c30460" containerName="main"
Sep 15 19:17:35.143382 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.143266    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0d9a2f8-04b4-4728-b786-68974d219708" containerName="maas-api"
Sep 15 19:17:35.143382 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.143271    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0d9a2f8-04b4-4728-b786-68974d219708" containerName="maas-api"
Sep 15 19:17:35.143382 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.143350    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d08b3386-95ee-42ab-8fd6-5787bed9336a" containerName="maas-api"
Sep 15 19:17:35.143382 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.143358    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5be5a39b-2af4-453b-8de6-06962e9cc44d" containerName="maas-api"
Sep 15 19:17:35.143382 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.143366    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e52edd-9587-47c1-b9bc-e8d7a7c30460" containerName="main"
Sep 15 19:17:35.143382 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.143372    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0d9a2f8-04b4-4728-b786-68974d219708" containerName="maas-api"
Sep 15 19:17:35.143382 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.143378    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd3216a3-8c92-4e1b-884a-bf5e90d1f772" containerName="maas-api"
Sep 15 19:17:35.143382 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.143384    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade125f2-e25c-40ad-86ae-66ac1bb33467" containerName="main"
Sep 15 19:17:35.146416 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.146399    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.148856 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.148838    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-33ff04\"/\"kube-root-ca.crt\""
Sep 15 19:17:35.149570 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.149550    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-33ff04\"/\"openshift-service-ca.crt\""
Sep 15 19:17:35.149682 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.149603    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-33ff04\"/\"rate-test-model-33ff04-kserve-self-signed-certs\""
Sep 15 19:17:35.149846 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.149817    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-33ff04\"/\"default-dockercfg-9prxg\""
Sep 15 19:17:35.157733 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.157712    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"]
Sep 15 19:17:35.199317 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.199287    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-dshm\") pod \"rate-test-model-33ff04-kserve-75f8ff5979-f546t\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.199504 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.199350    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpdp2\" (UniqueName: \"kubernetes.io/projected/1619de49-d55d-4a31-9de2-223330cccd93-kube-api-access-mpdp2\") pod \"rate-test-model-33ff04-kserve-75f8ff5979-f546t\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.199504 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.199377    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-tmp-dir\") pod \"rate-test-model-33ff04-kserve-75f8ff5979-f546t\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.199504 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.199415    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1619de49-d55d-4a31-9de2-223330cccd93-tls-certs\") pod \"rate-test-model-33ff04-kserve-75f8ff5979-f546t\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.199504 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.199464    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-model-cache\") pod \"rate-test-model-33ff04-kserve-75f8ff5979-f546t\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.199721 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.199543    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-home\") pod \"rate-test-model-33ff04-kserve-75f8ff5979-f546t\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.300209 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.300171    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-model-cache\") pod \"rate-test-model-33ff04-kserve-75f8ff5979-f546t\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.300409 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.300231    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-home\") pod \"rate-test-model-33ff04-kserve-75f8ff5979-f546t\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.300409 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.300260    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-dshm\") pod \"rate-test-model-33ff04-kserve-75f8ff5979-f546t\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.300409 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.300292    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpdp2\" (UniqueName: \"kubernetes.io/projected/1619de49-d55d-4a31-9de2-223330cccd93-kube-api-access-mpdp2\") pod \"rate-test-model-33ff04-kserve-75f8ff5979-f546t\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.300578 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.300429    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-tmp-dir\") pod \"rate-test-model-33ff04-kserve-75f8ff5979-f546t\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.300578 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.300496    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1619de49-d55d-4a31-9de2-223330cccd93-tls-certs\") pod \"rate-test-model-33ff04-kserve-75f8ff5979-f546t\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.300578 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.300567    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-model-cache\") pod \"rate-test-model-33ff04-kserve-75f8ff5979-f546t\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.300783 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.300661    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-home\") pod \"rate-test-model-33ff04-kserve-75f8ff5979-f546t\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.300826 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.300778    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-tmp-dir\") pod \"rate-test-model-33ff04-kserve-75f8ff5979-f546t\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.302600 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.302579    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-dshm\") pod \"rate-test-model-33ff04-kserve-75f8ff5979-f546t\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.302796 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.302780    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1619de49-d55d-4a31-9de2-223330cccd93-tls-certs\") pod \"rate-test-model-33ff04-kserve-75f8ff5979-f546t\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.309056 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.309036    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpdp2\" (UniqueName: \"kubernetes.io/projected/1619de49-d55d-4a31-9de2-223330cccd93-kube-api-access-mpdp2\") pod \"rate-test-model-33ff04-kserve-75f8ff5979-f546t\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") " pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.456429 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.456344    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:35.728133 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:35.728108    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"]
Sep 15 19:17:35.729515 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:17:35.729493    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1619de49_d55d_4a31_9de2_223330cccd93.slice/crio-7591136d039d53846ccc9d0a3041b07844cce6378d3d6b857f39b0f8c2f258c4 WatchSource:0}: Error finding container 7591136d039d53846ccc9d0a3041b07844cce6378d3d6b857f39b0f8c2f258c4: Status 404 returned error can't find the container with id 7591136d039d53846ccc9d0a3041b07844cce6378d3d6b857f39b0f8c2f258c4
Sep 15 19:17:36.373874 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:36.373840    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t" event={"ID":"1619de49-d55d-4a31-9de2-223330cccd93","Type":"ContainerStarted","Data":"a913f996ccc607bdfe5a5ce47bc05bc3f544aa1dc0efa3629c48abd25e1c7da1"}
Sep 15 19:17:36.373874 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:36.373879    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t" event={"ID":"1619de49-d55d-4a31-9de2-223330cccd93","Type":"ContainerStarted","Data":"7591136d039d53846ccc9d0a3041b07844cce6378d3d6b857f39b0f8c2f258c4"}
Sep 15 19:17:36.390683 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:36.390632    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t" podStartSLOduration=1.390600769 podStartE2EDuration="1.390600769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:17:36.388153217 +0000 UTC m=+1597.875482215" watchObservedRunningTime="2026-09-15 19:17:36.390600769 +0000 UTC m=+1597.877929806"
Sep 15 19:17:45.457164 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:45.457113    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:45.457688 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:45.457189    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:45.469795 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:45.469771    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:17:46.443381 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:17:46.443352    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:18:12.745344 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:12.745308    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"]
Sep 15 19:18:17.842371 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:17.842326    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"]
Sep 15 19:18:17.846590 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:17.846569    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"
Sep 15 19:18:17.848924 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:17.848901    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-6eb41d-serving-cert\""
Sep 15 19:18:17.855091 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:17.854680    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"]
Sep 15 19:18:17.893530 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:17.893488    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qcp8\" (UniqueName: \"kubernetes.io/projected/a23682a5-793e-4b1e-838f-f812f337ac40-kube-api-access-7qcp8\") pod \"maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt\" (UID: \"a23682a5-793e-4b1e-838f-f812f337ac40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"
Sep 15 19:18:17.893742 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:17.893539    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a23682a5-793e-4b1e-838f-f812f337ac40-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt\" (UID: \"a23682a5-793e-4b1e-838f-f812f337ac40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"
Sep 15 19:18:17.893742 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:17.893603    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a23682a5-793e-4b1e-838f-f812f337ac40-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt\" (UID: \"a23682a5-793e-4b1e-838f-f812f337ac40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"
Sep 15 19:18:17.893742 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:17.893640    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a23682a5-793e-4b1e-838f-f812f337ac40-metrics-certs\") pod \"maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt\" (UID: \"a23682a5-793e-4b1e-838f-f812f337ac40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"
Sep 15 19:18:17.994791 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:17.994752    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qcp8\" (UniqueName: \"kubernetes.io/projected/a23682a5-793e-4b1e-838f-f812f337ac40-kube-api-access-7qcp8\") pod \"maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt\" (UID: \"a23682a5-793e-4b1e-838f-f812f337ac40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"
Sep 15 19:18:17.994966 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:17.994810    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a23682a5-793e-4b1e-838f-f812f337ac40-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt\" (UID: \"a23682a5-793e-4b1e-838f-f812f337ac40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"
Sep 15 19:18:17.994966 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:17.994887    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a23682a5-793e-4b1e-838f-f812f337ac40-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt\" (UID: \"a23682a5-793e-4b1e-838f-f812f337ac40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"
Sep 15 19:18:17.994966 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:17.994907    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a23682a5-793e-4b1e-838f-f812f337ac40-metrics-certs\") pod \"maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt\" (UID: \"a23682a5-793e-4b1e-838f-f812f337ac40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"
Sep 15 19:18:17.994966 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:18:17.994912    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-serving-cert: secret "maas-api-e2e-ipp-a-6eb41d-serving-cert" not found
Sep 15 19:18:17.995134 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:18:17.994999    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a23682a5-793e-4b1e-838f-f812f337ac40-maas-api-tls podName:a23682a5-793e-4b1e-838f-f812f337ac40 nodeName:}" failed. No retries permitted until 2026-09-15 19:18:18.494981097 +0000 UTC m=+1639.982310059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a23682a5-793e-4b1e-838f-f812f337ac40-maas-api-tls") pod "maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt" (UID: "a23682a5-793e-4b1e-838f-f812f337ac40") : secret "maas-api-e2e-ipp-a-6eb41d-serving-cert" not found
Sep 15 19:18:17.995726 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:17.995700    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a23682a5-793e-4b1e-838f-f812f337ac40-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt\" (UID: \"a23682a5-793e-4b1e-838f-f812f337ac40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"
Sep 15 19:18:17.997436 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:17.997415    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a23682a5-793e-4b1e-838f-f812f337ac40-metrics-certs\") pod \"maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt\" (UID: \"a23682a5-793e-4b1e-838f-f812f337ac40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"
Sep 15 19:18:18.002427 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:18.002406    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qcp8\" (UniqueName: \"kubernetes.io/projected/a23682a5-793e-4b1e-838f-f812f337ac40-kube-api-access-7qcp8\") pod \"maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt\" (UID: \"a23682a5-793e-4b1e-838f-f812f337ac40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"
Sep 15 19:18:18.499854 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:18.499816    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a23682a5-793e-4b1e-838f-f812f337ac40-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt\" (UID: \"a23682a5-793e-4b1e-838f-f812f337ac40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"
Sep 15 19:18:18.502302 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:18.502278    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a23682a5-793e-4b1e-838f-f812f337ac40-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt\" (UID: \"a23682a5-793e-4b1e-838f-f812f337ac40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"
Sep 15 19:18:18.759296 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:18.759200    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"
Sep 15 19:18:19.036989 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:18:19.036961    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23682a5_793e_4b1e_838f_f812f337ac40.slice/crio-438f03312855df55b912501a615782f05c00aa7489789ee3078ad657c1f19ab1 WatchSource:0}: Error finding container 438f03312855df55b912501a615782f05c00aa7489789ee3078ad657c1f19ab1: Status 404 returned error can't find the container with id 438f03312855df55b912501a615782f05c00aa7489789ee3078ad657c1f19ab1
Sep 15 19:18:19.042757 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:19.042734    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"]
Sep 15 19:18:19.603934 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:19.603897    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt" event={"ID":"a23682a5-793e-4b1e-838f-f812f337ac40","Type":"ContainerStarted","Data":"71e87f474766d28b3708dffb4e898ab00f7a2589aad98a3bfbca66a9e67b202f"}
Sep 15 19:18:19.604103 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:19.603949    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt" event={"ID":"a23682a5-793e-4b1e-838f-f812f337ac40","Type":"ContainerStarted","Data":"438f03312855df55b912501a615782f05c00aa7489789ee3078ad657c1f19ab1"}
Sep 15 19:18:19.604103 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:19.604042    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"
Sep 15 19:18:19.621625 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:19.621559    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt" podStartSLOduration=2.166960356 podStartE2EDuration="2.621543675s" totalImagesPullingTime="454.583319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:18:19.619535732 +0000 UTC m=+1641.106864717" watchObservedRunningTime="2026-09-15 19:18:19.621543675 +0000 UTC m=+1641.108872659"
Sep 15 19:18:25.615476 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:25.615448    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"
Sep 15 19:18:27.764981 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:27.764936    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t" podUID="1619de49-d55d-4a31-9de2-223330cccd93" containerName="main" containerID="cri-o://a913f996ccc607bdfe5a5ce47bc05bc3f544aa1dc0efa3629c48abd25e1c7da1" gracePeriod=45
Sep 15 19:18:28.012753 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.012729    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:18:28.084794 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.084717    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpdp2\" (UniqueName: \"kubernetes.io/projected/1619de49-d55d-4a31-9de2-223330cccd93-kube-api-access-mpdp2\") pod \"1619de49-d55d-4a31-9de2-223330cccd93\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") "
Sep 15 19:18:28.084794 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.084756    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-model-cache\") pod \"1619de49-d55d-4a31-9de2-223330cccd93\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") "
Sep 15 19:18:28.085020 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.084823    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1619de49-d55d-4a31-9de2-223330cccd93-tls-certs\") pod \"1619de49-d55d-4a31-9de2-223330cccd93\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") "
Sep 15 19:18:28.085020 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.084850    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-dshm\") pod \"1619de49-d55d-4a31-9de2-223330cccd93\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") "
Sep 15 19:18:28.085020 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.084874    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-tmp-dir\") pod \"1619de49-d55d-4a31-9de2-223330cccd93\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") "
Sep 15 19:18:28.085020 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.084925    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-home\") pod \"1619de49-d55d-4a31-9de2-223330cccd93\" (UID: \"1619de49-d55d-4a31-9de2-223330cccd93\") "
Sep 15 19:18:28.085252 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.085073    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-model-cache" (OuterVolumeSpecName: "model-cache") pod "1619de49-d55d-4a31-9de2-223330cccd93" (UID: "1619de49-d55d-4a31-9de2-223330cccd93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:28.085252 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.085183    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1619de49-d55d-4a31-9de2-223330cccd93" (UID: "1619de49-d55d-4a31-9de2-223330cccd93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:28.085662 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.085639    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-tmp-dir\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:18:28.085786 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.085665    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-model-cache\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:18:28.085899 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.085873    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-home" (OuterVolumeSpecName: "home") pod "1619de49-d55d-4a31-9de2-223330cccd93" (UID: "1619de49-d55d-4a31-9de2-223330cccd93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:28.087017 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.086979    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1619de49-d55d-4a31-9de2-223330cccd93-kube-api-access-mpdp2" (OuterVolumeSpecName: "kube-api-access-mpdp2") pod "1619de49-d55d-4a31-9de2-223330cccd93" (UID: "1619de49-d55d-4a31-9de2-223330cccd93"). InnerVolumeSpecName "kube-api-access-mpdp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:18:28.087122 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.087049    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-dshm" (OuterVolumeSpecName: "dshm") pod "1619de49-d55d-4a31-9de2-223330cccd93" (UID: "1619de49-d55d-4a31-9de2-223330cccd93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:28.087122 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.087063    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1619de49-d55d-4a31-9de2-223330cccd93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1619de49-d55d-4a31-9de2-223330cccd93" (UID: "1619de49-d55d-4a31-9de2-223330cccd93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:18:28.186441 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.186406    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1619de49-d55d-4a31-9de2-223330cccd93-tls-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:18:28.186441 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.186437    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-dshm\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:18:28.186441 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.186446    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1619de49-d55d-4a31-9de2-223330cccd93-home\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:18:28.186658 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.186454    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpdp2\" (UniqueName: \"kubernetes.io/projected/1619de49-d55d-4a31-9de2-223330cccd93-kube-api-access-mpdp2\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:18:28.650808 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.650777    2555 generic.go:356] "Generic (PLEG): container finished" podID="1619de49-d55d-4a31-9de2-223330cccd93" containerID="a913f996ccc607bdfe5a5ce47bc05bc3f544aa1dc0efa3629c48abd25e1c7da1" exitCode=0
Sep 15 19:18:28.651022 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.650834    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"
Sep 15 19:18:28.651022 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.650871    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t" event={"ID":"1619de49-d55d-4a31-9de2-223330cccd93","Type":"ContainerDied","Data":"a913f996ccc607bdfe5a5ce47bc05bc3f544aa1dc0efa3629c48abd25e1c7da1"}
Sep 15 19:18:28.651022 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.650923    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t" event={"ID":"1619de49-d55d-4a31-9de2-223330cccd93","Type":"ContainerDied","Data":"7591136d039d53846ccc9d0a3041b07844cce6378d3d6b857f39b0f8c2f258c4"}
Sep 15 19:18:28.651022 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.650941    2555 scope.go:117] "RemoveContainer" containerID="a913f996ccc607bdfe5a5ce47bc05bc3f544aa1dc0efa3629c48abd25e1c7da1"
Sep 15 19:18:28.663209 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.663118    2555 scope.go:117] "RemoveContainer" containerID="a913f996ccc607bdfe5a5ce47bc05bc3f544aa1dc0efa3629c48abd25e1c7da1"
Sep 15 19:18:28.663784 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:18:28.663752    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a913f996ccc607bdfe5a5ce47bc05bc3f544aa1dc0efa3629c48abd25e1c7da1\": container with ID starting with a913f996ccc607bdfe5a5ce47bc05bc3f544aa1dc0efa3629c48abd25e1c7da1 not found: ID does not exist" containerID="a913f996ccc607bdfe5a5ce47bc05bc3f544aa1dc0efa3629c48abd25e1c7da1"
Sep 15 19:18:28.663896 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.663789    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a913f996ccc607bdfe5a5ce47bc05bc3f544aa1dc0efa3629c48abd25e1c7da1"} err="failed to get container status \"a913f996ccc607bdfe5a5ce47bc05bc3f544aa1dc0efa3629c48abd25e1c7da1\": rpc error: code = NotFound desc = could not find container \"a913f996ccc607bdfe5a5ce47bc05bc3f544aa1dc0efa3629c48abd25e1c7da1\": container with ID starting with a913f996ccc607bdfe5a5ce47bc05bc3f544aa1dc0efa3629c48abd25e1c7da1 not found: ID does not exist"
Sep 15 19:18:28.675526 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.675492    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"]
Sep 15 19:18:28.678956 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:28.678933    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-33ff04/rate-test-model-33ff04-kserve-75f8ff5979-f546t"]
Sep 15 19:18:29.041007 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:29.040969    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1619de49-d55d-4a31-9de2-223330cccd93" path="/var/lib/kubelet/pods/1619de49-d55d-4a31-9de2-223330cccd93/volumes"
Sep 15 19:18:47.732637 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:47.732585    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"]
Sep 15 19:18:47.733186 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:47.733151    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1619de49-d55d-4a31-9de2-223330cccd93" containerName="main"
Sep 15 19:18:47.733186 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:47.733177    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1619de49-d55d-4a31-9de2-223330cccd93" containerName="main"
Sep 15 19:18:47.733395 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:47.733274    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1619de49-d55d-4a31-9de2-223330cccd93" containerName="main"
Sep 15 19:18:47.736426 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:47.736408    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"
Sep 15 19:18:47.738696 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:47.738673    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-25b1b3-serving-cert\""
Sep 15 19:18:47.748701 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:47.748668    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"]
Sep 15 19:18:47.875186 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:47.875152    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65819aa2-b0f6-48e2-9c17-1a8f35722773-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh\" (UID: \"65819aa2-b0f6-48e2-9c17-1a8f35722773\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"
Sep 15 19:18:47.875355 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:47.875215    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnpdg\" (UniqueName: \"kubernetes.io/projected/65819aa2-b0f6-48e2-9c17-1a8f35722773-kube-api-access-nnpdg\") pod \"maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh\" (UID: \"65819aa2-b0f6-48e2-9c17-1a8f35722773\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"
Sep 15 19:18:47.875355 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:47.875336    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65819aa2-b0f6-48e2-9c17-1a8f35722773-metrics-certs\") pod \"maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh\" (UID: \"65819aa2-b0f6-48e2-9c17-1a8f35722773\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"
Sep 15 19:18:47.875431 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:47.875391    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65819aa2-b0f6-48e2-9c17-1a8f35722773-maas-api-tls\") pod \"maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh\" (UID: \"65819aa2-b0f6-48e2-9c17-1a8f35722773\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"
Sep 15 19:18:47.976922 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:47.976887    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnpdg\" (UniqueName: \"kubernetes.io/projected/65819aa2-b0f6-48e2-9c17-1a8f35722773-kube-api-access-nnpdg\") pod \"maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh\" (UID: \"65819aa2-b0f6-48e2-9c17-1a8f35722773\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"
Sep 15 19:18:47.977095 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:47.976945    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65819aa2-b0f6-48e2-9c17-1a8f35722773-metrics-certs\") pod \"maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh\" (UID: \"65819aa2-b0f6-48e2-9c17-1a8f35722773\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"
Sep 15 19:18:47.977095 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:47.976973    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65819aa2-b0f6-48e2-9c17-1a8f35722773-maas-api-tls\") pod \"maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh\" (UID: \"65819aa2-b0f6-48e2-9c17-1a8f35722773\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"
Sep 15 19:18:47.977095 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:47.977007    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65819aa2-b0f6-48e2-9c17-1a8f35722773-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh\" (UID: \"65819aa2-b0f6-48e2-9c17-1a8f35722773\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"
Sep 15 19:18:47.977598 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:47.977580    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65819aa2-b0f6-48e2-9c17-1a8f35722773-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh\" (UID: \"65819aa2-b0f6-48e2-9c17-1a8f35722773\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"
Sep 15 19:18:47.979539 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:47.979521    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65819aa2-b0f6-48e2-9c17-1a8f35722773-maas-api-tls\") pod \"maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh\" (UID: \"65819aa2-b0f6-48e2-9c17-1a8f35722773\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"
Sep 15 19:18:47.979602 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:47.979555    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65819aa2-b0f6-48e2-9c17-1a8f35722773-metrics-certs\") pod \"maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh\" (UID: \"65819aa2-b0f6-48e2-9c17-1a8f35722773\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"
Sep 15 19:18:47.986502 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:47.986441    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnpdg\" (UniqueName: \"kubernetes.io/projected/65819aa2-b0f6-48e2-9c17-1a8f35722773-kube-api-access-nnpdg\") pod \"maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh\" (UID: \"65819aa2-b0f6-48e2-9c17-1a8f35722773\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"
Sep 15 19:18:48.050322 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:48.050290    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"
Sep 15 19:18:48.325497 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:48.325473    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"]
Sep 15 19:18:48.327655 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:18:48.327628    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65819aa2_b0f6_48e2_9c17_1a8f35722773.slice/crio-a4832f473c589e045ac96b6ea300cfe450f9a634172551e51eb2aa7a025144b5 WatchSource:0}: Error finding container a4832f473c589e045ac96b6ea300cfe450f9a634172551e51eb2aa7a025144b5: Status 404 returned error can't find the container with id a4832f473c589e045ac96b6ea300cfe450f9a634172551e51eb2aa7a025144b5
Sep 15 19:18:48.763441 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:48.763398    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh" event={"ID":"65819aa2-b0f6-48e2-9c17-1a8f35722773","Type":"ContainerStarted","Data":"a4832f473c589e045ac96b6ea300cfe450f9a634172551e51eb2aa7a025144b5"}
Sep 15 19:18:49.770551 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:49.770519    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh" event={"ID":"65819aa2-b0f6-48e2-9c17-1a8f35722773","Type":"ContainerStarted","Data":"9af6f09a3f75517f57f9278c523f1c9ba10132e7abae92c6200183474659bd15"}
Sep 15 19:18:49.770959 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:49.770579    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"
Sep 15 19:18:49.785757 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:49.785671    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh" podStartSLOduration=2.28819827 podStartE2EDuration="2.785651472s" totalImagesPullingTime="497.453202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:18:49.785040709 +0000 UTC m=+1671.272369696" watchObservedRunningTime="2026-09-15 19:18:49.785651472 +0000 UTC m=+1671.272980459"
Sep 15 19:18:55.781661 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:18:55.781630    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"
Sep 15 19:19:18.934298 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:18.934265    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"]
Sep 15 19:19:18.938167 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:18.938144    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:18.941662 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:18.941635    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6eb41d\"/\"default-dockercfg-w6c99\""
Sep 15 19:19:18.941813 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:18.941659    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6eb41d\"/\"ipp-route-6eb41d-kserve-self-signed-certs\""
Sep 15 19:19:18.941813 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:18.941672    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6eb41d\"/\"kube-root-ca.crt\""
Sep 15 19:19:18.941813 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:18.941635    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6eb41d\"/\"openshift-service-ca.crt\""
Sep 15 19:19:18.948302 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:18.948277    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"]
Sep 15 19:19:19.072975 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.072932    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-tls-certs\") pod \"ipp-route-6eb41d-kserve-765f5b876c-ltljc\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") " pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:19.073200 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.072999    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-tmp-dir\") pod \"ipp-route-6eb41d-kserve-765f5b876c-ltljc\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") " pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:19.073200 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.073037    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-dshm\") pod \"ipp-route-6eb41d-kserve-765f5b876c-ltljc\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") " pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:19.073200 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.073070    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-home\") pod \"ipp-route-6eb41d-kserve-765f5b876c-ltljc\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") " pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:19.073200 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.073124    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-model-cache\") pod \"ipp-route-6eb41d-kserve-765f5b876c-ltljc\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") " pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:19.073200 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.073195    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxpvw\" (UniqueName: \"kubernetes.io/projected/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-kube-api-access-cxpvw\") pod \"ipp-route-6eb41d-kserve-765f5b876c-ltljc\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") " pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:19.173975 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.173939    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-model-cache\") pod \"ipp-route-6eb41d-kserve-765f5b876c-ltljc\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") " pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:19.174132 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.173999    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxpvw\" (UniqueName: \"kubernetes.io/projected/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-kube-api-access-cxpvw\") pod \"ipp-route-6eb41d-kserve-765f5b876c-ltljc\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") " pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:19.174132 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.174083    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-tls-certs\") pod \"ipp-route-6eb41d-kserve-765f5b876c-ltljc\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") " pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:19.174132 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.174126    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-tmp-dir\") pod \"ipp-route-6eb41d-kserve-765f5b876c-ltljc\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") " pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:19.174278 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.174164    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-dshm\") pod \"ipp-route-6eb41d-kserve-765f5b876c-ltljc\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") " pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:19.174278 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.174221    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-home\") pod \"ipp-route-6eb41d-kserve-765f5b876c-ltljc\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") " pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:19.174416 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.174395    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-model-cache\") pod \"ipp-route-6eb41d-kserve-765f5b876c-ltljc\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") " pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:19.174513 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.174492    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-tmp-dir\") pod \"ipp-route-6eb41d-kserve-765f5b876c-ltljc\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") " pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:19.174622 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.174577    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-home\") pod \"ipp-route-6eb41d-kserve-765f5b876c-ltljc\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") " pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:19.176401 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.176385    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-dshm\") pod \"ipp-route-6eb41d-kserve-765f5b876c-ltljc\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") " pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:19.176511 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.176495    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-tls-certs\") pod \"ipp-route-6eb41d-kserve-765f5b876c-ltljc\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") " pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:19.181438 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.181416    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxpvw\" (UniqueName: \"kubernetes.io/projected/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-kube-api-access-cxpvw\") pod \"ipp-route-6eb41d-kserve-765f5b876c-ltljc\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") " pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:19.248790 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.248758    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:19.523320 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.523290    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"]
Sep 15 19:19:19.524927 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:19:19.524888    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4edb8dc8_4604_41c3_bca6_2c38fa97c6dd.slice/crio-47ee217ae6d3d3010a79939b0cf792e721b6ffbf149c969f59a3713b8a59a523 WatchSource:0}: Error finding container 47ee217ae6d3d3010a79939b0cf792e721b6ffbf149c969f59a3713b8a59a523: Status 404 returned error can't find the container with id 47ee217ae6d3d3010a79939b0cf792e721b6ffbf149c969f59a3713b8a59a523
Sep 15 19:19:19.922569 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.922474    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc" event={"ID":"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd","Type":"ContainerStarted","Data":"708d010c05f5b52922bc8e2404986149d65de6d776a92da959333028a40d8ea1"}
Sep 15 19:19:19.922569 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.922526    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc" event={"ID":"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd","Type":"ContainerStarted","Data":"47ee217ae6d3d3010a79939b0cf792e721b6ffbf149c969f59a3713b8a59a523"}
Sep 15 19:19:19.939630 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:19.939549    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc" podStartSLOduration=1.939530881 podStartE2EDuration="1.939530881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:19:19.938820844 +0000 UTC m=+1701.426149843" watchObservedRunningTime="2026-09-15 19:19:19.939530881 +0000 UTC m=+1701.426859872"
Sep 15 19:19:29.249159 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:29.249106    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:29.249159 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:29.249169    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:29.261824 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:29.261794    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:29.986299 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:29.986269    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:19:54.468189 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:54.468140    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"]
Sep 15 19:19:54.472893 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:54.472870    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"
Sep 15 19:19:54.474924 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:54.474903    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-93b6ef-serving-cert\""
Sep 15 19:19:54.482959 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:54.482939    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"]
Sep 15 19:19:54.525419 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:54.525360    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc57f241-7447-4f92-874b-df716f1e5b05-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr\" (UID: \"cc57f241-7447-4f92-874b-df716f1e5b05\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"
Sep 15 19:19:54.525596 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:54.525501    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc57f241-7447-4f92-874b-df716f1e5b05-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr\" (UID: \"cc57f241-7447-4f92-874b-df716f1e5b05\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"
Sep 15 19:19:54.525596 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:54.525537    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns2vk\" (UniqueName: \"kubernetes.io/projected/cc57f241-7447-4f92-874b-df716f1e5b05-kube-api-access-ns2vk\") pod \"maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr\" (UID: \"cc57f241-7447-4f92-874b-df716f1e5b05\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"
Sep 15 19:19:54.525596 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:54.525584    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc57f241-7447-4f92-874b-df716f1e5b05-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr\" (UID: \"cc57f241-7447-4f92-874b-df716f1e5b05\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"
Sep 15 19:19:54.626534 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:54.626497    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc57f241-7447-4f92-874b-df716f1e5b05-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr\" (UID: \"cc57f241-7447-4f92-874b-df716f1e5b05\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"
Sep 15 19:19:54.626534 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:54.626538    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns2vk\" (UniqueName: \"kubernetes.io/projected/cc57f241-7447-4f92-874b-df716f1e5b05-kube-api-access-ns2vk\") pod \"maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr\" (UID: \"cc57f241-7447-4f92-874b-df716f1e5b05\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"
Sep 15 19:19:54.626815 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:54.626594    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc57f241-7447-4f92-874b-df716f1e5b05-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr\" (UID: \"cc57f241-7447-4f92-874b-df716f1e5b05\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"
Sep 15 19:19:54.626815 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:54.626674    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc57f241-7447-4f92-874b-df716f1e5b05-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr\" (UID: \"cc57f241-7447-4f92-874b-df716f1e5b05\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"
Sep 15 19:19:54.627154 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:54.627133    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc57f241-7447-4f92-874b-df716f1e5b05-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr\" (UID: \"cc57f241-7447-4f92-874b-df716f1e5b05\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"
Sep 15 19:19:54.628988 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:54.628960    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc57f241-7447-4f92-874b-df716f1e5b05-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr\" (UID: \"cc57f241-7447-4f92-874b-df716f1e5b05\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"
Sep 15 19:19:54.629200 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:54.629179    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc57f241-7447-4f92-874b-df716f1e5b05-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr\" (UID: \"cc57f241-7447-4f92-874b-df716f1e5b05\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"
Sep 15 19:19:54.635376 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:54.635352    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns2vk\" (UniqueName: \"kubernetes.io/projected/cc57f241-7447-4f92-874b-df716f1e5b05-kube-api-access-ns2vk\") pod \"maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr\" (UID: \"cc57f241-7447-4f92-874b-df716f1e5b05\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"
Sep 15 19:19:54.787554 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:54.787519    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"
Sep 15 19:19:55.085940 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:55.085855    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"]
Sep 15 19:19:55.089111 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:19:55.089088    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc57f241_7447_4f92_874b_df716f1e5b05.slice/crio-72575361d6b281d14da3c89f1fc67caaa07a22e59dd498a4b9550af6c372f828 WatchSource:0}: Error finding container 72575361d6b281d14da3c89f1fc67caaa07a22e59dd498a4b9550af6c372f828: Status 404 returned error can't find the container with id 72575361d6b281d14da3c89f1fc67caaa07a22e59dd498a4b9550af6c372f828
Sep 15 19:19:55.107642 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:55.107598    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr" event={"ID":"cc57f241-7447-4f92-874b-df716f1e5b05","Type":"ContainerStarted","Data":"72575361d6b281d14da3c89f1fc67caaa07a22e59dd498a4b9550af6c372f828"}
Sep 15 19:19:56.113956 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:56.113923    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr" event={"ID":"cc57f241-7447-4f92-874b-df716f1e5b05","Type":"ContainerStarted","Data":"f854df51e81d559bed22f8194cbf1e7fb22319898ac895518533e949c619ff30"}
Sep 15 19:19:56.114381 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:56.114162    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"
Sep 15 19:19:56.131671 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:19:56.131622    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr" podStartSLOduration=1.8126389710000002 podStartE2EDuration="2.131596239s" totalImagesPullingTime="318.957268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:19:56.129603213 +0000 UTC m=+1737.616932200" watchObservedRunningTime="2026-09-15 19:19:56.131596239 +0000 UTC m=+1737.618925223"
Sep 15 19:20:02.124364 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:02.124334    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"
Sep 15 19:20:23.558714 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:23.558675    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"]
Sep 15 19:20:23.559295 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:23.558976    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr" podUID="cc57f241-7447-4f92-874b-df716f1e5b05" containerName="maas-api" containerID="cri-o://f854df51e81d559bed22f8194cbf1e7fb22319898ac895518533e949c619ff30" gracePeriod=30
Sep 15 19:20:23.801268 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:23.801244    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"
Sep 15 19:20:23.905436 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:23.905337    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns2vk\" (UniqueName: \"kubernetes.io/projected/cc57f241-7447-4f92-874b-df716f1e5b05-kube-api-access-ns2vk\") pod \"cc57f241-7447-4f92-874b-df716f1e5b05\" (UID: \"cc57f241-7447-4f92-874b-df716f1e5b05\") "
Sep 15 19:20:23.905642 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:23.905450    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc57f241-7447-4f92-874b-df716f1e5b05-metrics-certs\") pod \"cc57f241-7447-4f92-874b-df716f1e5b05\" (UID: \"cc57f241-7447-4f92-874b-df716f1e5b05\") "
Sep 15 19:20:23.905642 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:23.905481    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc57f241-7447-4f92-874b-df716f1e5b05-openshift-service-ca\") pod \"cc57f241-7447-4f92-874b-df716f1e5b05\" (UID: \"cc57f241-7447-4f92-874b-df716f1e5b05\") "
Sep 15 19:20:23.905642 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:23.905530    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc57f241-7447-4f92-874b-df716f1e5b05-maas-api-tls\") pod \"cc57f241-7447-4f92-874b-df716f1e5b05\" (UID: \"cc57f241-7447-4f92-874b-df716f1e5b05\") "
Sep 15 19:20:23.905957 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:23.905917    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc57f241-7447-4f92-874b-df716f1e5b05-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc57f241-7447-4f92-874b-df716f1e5b05" (UID: "cc57f241-7447-4f92-874b-df716f1e5b05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:20:23.907823 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:23.907793    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc57f241-7447-4f92-874b-df716f1e5b05-kube-api-access-ns2vk" (OuterVolumeSpecName: "kube-api-access-ns2vk") pod "cc57f241-7447-4f92-874b-df716f1e5b05" (UID: "cc57f241-7447-4f92-874b-df716f1e5b05"). InnerVolumeSpecName "kube-api-access-ns2vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:20:23.907823 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:23.907807    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc57f241-7447-4f92-874b-df716f1e5b05-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cc57f241-7447-4f92-874b-df716f1e5b05" (UID: "cc57f241-7447-4f92-874b-df716f1e5b05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:20:23.908022 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:23.908001    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc57f241-7447-4f92-874b-df716f1e5b05-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc57f241-7447-4f92-874b-df716f1e5b05" (UID: "cc57f241-7447-4f92-874b-df716f1e5b05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:20:24.006428 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:24.006392    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc57f241-7447-4f92-874b-df716f1e5b05-maas-api-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:20:24.006428 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:24.006422    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns2vk\" (UniqueName: \"kubernetes.io/projected/cc57f241-7447-4f92-874b-df716f1e5b05-kube-api-access-ns2vk\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:20:24.006428 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:24.006433    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc57f241-7447-4f92-874b-df716f1e5b05-metrics-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:20:24.006689 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:24.006441    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc57f241-7447-4f92-874b-df716f1e5b05-openshift-service-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:20:24.266128 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:24.266095    2555 generic.go:356] "Generic (PLEG): container finished" podID="cc57f241-7447-4f92-874b-df716f1e5b05" containerID="f854df51e81d559bed22f8194cbf1e7fb22319898ac895518533e949c619ff30" exitCode=0
Sep 15 19:20:24.266299 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:24.266141    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"
Sep 15 19:20:24.266299 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:24.266181    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr" event={"ID":"cc57f241-7447-4f92-874b-df716f1e5b05","Type":"ContainerDied","Data":"f854df51e81d559bed22f8194cbf1e7fb22319898ac895518533e949c619ff30"}
Sep 15 19:20:24.266299 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:24.266229    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr" event={"ID":"cc57f241-7447-4f92-874b-df716f1e5b05","Type":"ContainerDied","Data":"72575361d6b281d14da3c89f1fc67caaa07a22e59dd498a4b9550af6c372f828"}
Sep 15 19:20:24.266299 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:24.266257    2555 scope.go:117] "RemoveContainer" containerID="f854df51e81d559bed22f8194cbf1e7fb22319898ac895518533e949c619ff30"
Sep 15 19:20:24.276094 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:24.276080    2555 scope.go:117] "RemoveContainer" containerID="f854df51e81d559bed22f8194cbf1e7fb22319898ac895518533e949c619ff30"
Sep 15 19:20:24.276349 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:20:24.276333    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f854df51e81d559bed22f8194cbf1e7fb22319898ac895518533e949c619ff30\": container with ID starting with f854df51e81d559bed22f8194cbf1e7fb22319898ac895518533e949c619ff30 not found: ID does not exist" containerID="f854df51e81d559bed22f8194cbf1e7fb22319898ac895518533e949c619ff30"
Sep 15 19:20:24.276447 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:24.276356    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f854df51e81d559bed22f8194cbf1e7fb22319898ac895518533e949c619ff30"} err="failed to get container status \"f854df51e81d559bed22f8194cbf1e7fb22319898ac895518533e949c619ff30\": rpc error: code = NotFound desc = could not find container \"f854df51e81d559bed22f8194cbf1e7fb22319898ac895518533e949c619ff30\": container with ID starting with f854df51e81d559bed22f8194cbf1e7fb22319898ac895518533e949c619ff30 not found: ID does not exist"
Sep 15 19:20:24.287276 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:24.287240    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"]
Sep 15 19:20:24.289032 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:24.289013    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-93b6ef-8dc9b5df9-n97fr"]
Sep 15 19:20:25.040839 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:25.040809    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc57f241-7447-4f92-874b-df716f1e5b05" path="/var/lib/kubelet/pods/cc57f241-7447-4f92-874b-df716f1e5b05/volumes"
Sep 15 19:20:49.494276 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:49.494244    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"]
Sep 15 19:20:49.494869 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:49.494495    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt" podUID="a23682a5-793e-4b1e-838f-f812f337ac40" containerName="maas-api" containerID="cri-o://71e87f474766d28b3708dffb4e898ab00f7a2589aad98a3bfbca66a9e67b202f" gracePeriod=30
Sep 15 19:20:49.742155 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:49.742129    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"
Sep 15 19:20:49.860991 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:49.860902    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a23682a5-793e-4b1e-838f-f812f337ac40-openshift-service-ca\") pod \"a23682a5-793e-4b1e-838f-f812f337ac40\" (UID: \"a23682a5-793e-4b1e-838f-f812f337ac40\") "
Sep 15 19:20:49.861223 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:49.861168    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a23682a5-793e-4b1e-838f-f812f337ac40-maas-api-tls\") pod \"a23682a5-793e-4b1e-838f-f812f337ac40\" (UID: \"a23682a5-793e-4b1e-838f-f812f337ac40\") "
Sep 15 19:20:49.861312 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:49.861233    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qcp8\" (UniqueName: \"kubernetes.io/projected/a23682a5-793e-4b1e-838f-f812f337ac40-kube-api-access-7qcp8\") pod \"a23682a5-793e-4b1e-838f-f812f337ac40\" (UID: \"a23682a5-793e-4b1e-838f-f812f337ac40\") "
Sep 15 19:20:49.861312 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:49.861279    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a23682a5-793e-4b1e-838f-f812f337ac40-metrics-certs\") pod \"a23682a5-793e-4b1e-838f-f812f337ac40\" (UID: \"a23682a5-793e-4b1e-838f-f812f337ac40\") "
Sep 15 19:20:49.861417 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:49.861336    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23682a5-793e-4b1e-838f-f812f337ac40-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a23682a5-793e-4b1e-838f-f812f337ac40" (UID: "a23682a5-793e-4b1e-838f-f812f337ac40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:20:49.861685 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:49.861658    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a23682a5-793e-4b1e-838f-f812f337ac40-openshift-service-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:20:49.863573 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:49.863536    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23682a5-793e-4b1e-838f-f812f337ac40-kube-api-access-7qcp8" (OuterVolumeSpecName: "kube-api-access-7qcp8") pod "a23682a5-793e-4b1e-838f-f812f337ac40" (UID: "a23682a5-793e-4b1e-838f-f812f337ac40"). InnerVolumeSpecName "kube-api-access-7qcp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:20:49.863832 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:49.863811    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a23682a5-793e-4b1e-838f-f812f337ac40-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a23682a5-793e-4b1e-838f-f812f337ac40" (UID: "a23682a5-793e-4b1e-838f-f812f337ac40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:20:49.863902 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:49.863874    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a23682a5-793e-4b1e-838f-f812f337ac40-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a23682a5-793e-4b1e-838f-f812f337ac40" (UID: "a23682a5-793e-4b1e-838f-f812f337ac40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:20:49.962538 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:49.962506    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a23682a5-793e-4b1e-838f-f812f337ac40-maas-api-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:20:49.962538 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:49.962534    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qcp8\" (UniqueName: \"kubernetes.io/projected/a23682a5-793e-4b1e-838f-f812f337ac40-kube-api-access-7qcp8\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:20:49.962538 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:49.962545    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a23682a5-793e-4b1e-838f-f812f337ac40-metrics-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:20:50.402955 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:50.402922    2555 generic.go:356] "Generic (PLEG): container finished" podID="a23682a5-793e-4b1e-838f-f812f337ac40" containerID="71e87f474766d28b3708dffb4e898ab00f7a2589aad98a3bfbca66a9e67b202f" exitCode=0
Sep 15 19:20:50.403149 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:50.402964    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt" event={"ID":"a23682a5-793e-4b1e-838f-f812f337ac40","Type":"ContainerDied","Data":"71e87f474766d28b3708dffb4e898ab00f7a2589aad98a3bfbca66a9e67b202f"}
Sep 15 19:20:50.403149 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:50.402975    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"
Sep 15 19:20:50.403149 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:50.403004    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt" event={"ID":"a23682a5-793e-4b1e-838f-f812f337ac40","Type":"ContainerDied","Data":"438f03312855df55b912501a615782f05c00aa7489789ee3078ad657c1f19ab1"}
Sep 15 19:20:50.403149 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:50.403026    2555 scope.go:117] "RemoveContainer" containerID="71e87f474766d28b3708dffb4e898ab00f7a2589aad98a3bfbca66a9e67b202f"
Sep 15 19:20:50.413817 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:50.413799    2555 scope.go:117] "RemoveContainer" containerID="71e87f474766d28b3708dffb4e898ab00f7a2589aad98a3bfbca66a9e67b202f"
Sep 15 19:20:50.414115 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:20:50.414091    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e87f474766d28b3708dffb4e898ab00f7a2589aad98a3bfbca66a9e67b202f\": container with ID starting with 71e87f474766d28b3708dffb4e898ab00f7a2589aad98a3bfbca66a9e67b202f not found: ID does not exist" containerID="71e87f474766d28b3708dffb4e898ab00f7a2589aad98a3bfbca66a9e67b202f"
Sep 15 19:20:50.414177 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:50.414128    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e87f474766d28b3708dffb4e898ab00f7a2589aad98a3bfbca66a9e67b202f"} err="failed to get container status \"71e87f474766d28b3708dffb4e898ab00f7a2589aad98a3bfbca66a9e67b202f\": rpc error: code = NotFound desc = could not find container \"71e87f474766d28b3708dffb4e898ab00f7a2589aad98a3bfbca66a9e67b202f\": container with ID starting with 71e87f474766d28b3708dffb4e898ab00f7a2589aad98a3bfbca66a9e67b202f not found: ID does not exist"
Sep 15 19:20:50.425645 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:50.425619    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"]
Sep 15 19:20:50.431823 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:50.431801    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6eb41d-77d7f96779-s59pt"]
Sep 15 19:20:51.040861 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:51.040826    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a23682a5-793e-4b1e-838f-f812f337ac40" path="/var/lib/kubelet/pods/a23682a5-793e-4b1e-838f-f812f337ac40/volumes"
Sep 15 19:20:55.211495 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:20:55.211457    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"]
Sep 15 19:21:10.231774 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.231712    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc" podUID="4edb8dc8-4604-41c3-bca6-2c38fa97c6dd" containerName="main" containerID="cri-o://708d010c05f5b52922bc8e2404986149d65de6d776a92da959333028a40d8ea1" gracePeriod=45
Sep 15 19:21:10.482414 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.482354    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:21:10.515257 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.515222    2555 generic.go:356] "Generic (PLEG): container finished" podID="4edb8dc8-4604-41c3-bca6-2c38fa97c6dd" containerID="708d010c05f5b52922bc8e2404986149d65de6d776a92da959333028a40d8ea1" exitCode=0
Sep 15 19:21:10.515431 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.515318    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"
Sep 15 19:21:10.515431 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.515359    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc" event={"ID":"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd","Type":"ContainerDied","Data":"708d010c05f5b52922bc8e2404986149d65de6d776a92da959333028a40d8ea1"}
Sep 15 19:21:10.515431 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.515400    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc" event={"ID":"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd","Type":"ContainerDied","Data":"47ee217ae6d3d3010a79939b0cf792e721b6ffbf149c969f59a3713b8a59a523"}
Sep 15 19:21:10.515431 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.515419    2555 scope.go:117] "RemoveContainer" containerID="708d010c05f5b52922bc8e2404986149d65de6d776a92da959333028a40d8ea1"
Sep 15 19:21:10.525627 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.525589    2555 scope.go:117] "RemoveContainer" containerID="708d010c05f5b52922bc8e2404986149d65de6d776a92da959333028a40d8ea1"
Sep 15 19:21:10.525968 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:21:10.525944    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"708d010c05f5b52922bc8e2404986149d65de6d776a92da959333028a40d8ea1\": container with ID starting with 708d010c05f5b52922bc8e2404986149d65de6d776a92da959333028a40d8ea1 not found: ID does not exist" containerID="708d010c05f5b52922bc8e2404986149d65de6d776a92da959333028a40d8ea1"
Sep 15 19:21:10.526057 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.525978    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"708d010c05f5b52922bc8e2404986149d65de6d776a92da959333028a40d8ea1"} err="failed to get container status \"708d010c05f5b52922bc8e2404986149d65de6d776a92da959333028a40d8ea1\": rpc error: code = NotFound desc = could not find container \"708d010c05f5b52922bc8e2404986149d65de6d776a92da959333028a40d8ea1\": container with ID starting with 708d010c05f5b52922bc8e2404986149d65de6d776a92da959333028a40d8ea1 not found: ID does not exist"
Sep 15 19:21:10.556829 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.556801    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-tls-certs\") pod \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") "
Sep 15 19:21:10.556985 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.556865    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-tmp-dir\") pod \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") "
Sep 15 19:21:10.556985 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.556938    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-home\") pod \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") "
Sep 15 19:21:10.557085 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.556990    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-model-cache\") pod \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") "
Sep 15 19:21:10.557085 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.557034    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-dshm\") pod \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") "
Sep 15 19:21:10.557085 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.557076    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxpvw\" (UniqueName: \"kubernetes.io/projected/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-kube-api-access-cxpvw\") pod \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\" (UID: \"4edb8dc8-4604-41c3-bca6-2c38fa97c6dd\") "
Sep 15 19:21:10.557243 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.557180    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4edb8dc8-4604-41c3-bca6-2c38fa97c6dd" (UID: "4edb8dc8-4604-41c3-bca6-2c38fa97c6dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:10.557306 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.557231    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-home" (OuterVolumeSpecName: "home") pod "4edb8dc8-4604-41c3-bca6-2c38fa97c6dd" (UID: "4edb8dc8-4604-41c3-bca6-2c38fa97c6dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:10.557451 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.557419    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "4edb8dc8-4604-41c3-bca6-2c38fa97c6dd" (UID: "4edb8dc8-4604-41c3-bca6-2c38fa97c6dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:10.557620 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.557574    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-tmp-dir\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:21:10.557620 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.557589    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-home\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:21:10.557620 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.557598    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-model-cache\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:21:10.559287 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.559261    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-kube-api-access-cxpvw" (OuterVolumeSpecName: "kube-api-access-cxpvw") pod "4edb8dc8-4604-41c3-bca6-2c38fa97c6dd" (UID: "4edb8dc8-4604-41c3-bca6-2c38fa97c6dd"). InnerVolumeSpecName "kube-api-access-cxpvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:10.559446 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.559421    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-dshm" (OuterVolumeSpecName: "dshm") pod "4edb8dc8-4604-41c3-bca6-2c38fa97c6dd" (UID: "4edb8dc8-4604-41c3-bca6-2c38fa97c6dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:10.559521 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.559421    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4edb8dc8-4604-41c3-bca6-2c38fa97c6dd" (UID: "4edb8dc8-4604-41c3-bca6-2c38fa97c6dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:21:10.658957 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.658919    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-dshm\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:21:10.658957 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.658951    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxpvw\" (UniqueName: \"kubernetes.io/projected/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-kube-api-access-cxpvw\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:21:10.658957 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.658962    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd-tls-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:21:10.836449 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.836421    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"]
Sep 15 19:21:10.839943 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:10.839923    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-6eb41d/ipp-route-6eb41d-kserve-765f5b876c-ltljc"]
Sep 15 19:21:11.044789 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:11.041453    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4edb8dc8-4604-41c3-bca6-2c38fa97c6dd" path="/var/lib/kubelet/pods/4edb8dc8-4604-41c3-bca6-2c38fa97c6dd/volumes"
Sep 15 19:21:49.133342 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.133302    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"]
Sep 15 19:21:49.135935 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.133526    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh" podUID="65819aa2-b0f6-48e2-9c17-1a8f35722773" containerName="maas-api" containerID="cri-o://9af6f09a3f75517f57f9278c523f1c9ba10132e7abae92c6200183474659bd15" gracePeriod=30
Sep 15 19:21:49.379166 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.379141    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"
Sep 15 19:21:49.446822 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.446738    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnpdg\" (UniqueName: \"kubernetes.io/projected/65819aa2-b0f6-48e2-9c17-1a8f35722773-kube-api-access-nnpdg\") pod \"65819aa2-b0f6-48e2-9c17-1a8f35722773\" (UID: \"65819aa2-b0f6-48e2-9c17-1a8f35722773\") "
Sep 15 19:21:49.446822 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.446811    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65819aa2-b0f6-48e2-9c17-1a8f35722773-openshift-service-ca\") pod \"65819aa2-b0f6-48e2-9c17-1a8f35722773\" (UID: \"65819aa2-b0f6-48e2-9c17-1a8f35722773\") "
Sep 15 19:21:49.447055 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.446887    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65819aa2-b0f6-48e2-9c17-1a8f35722773-maas-api-tls\") pod \"65819aa2-b0f6-48e2-9c17-1a8f35722773\" (UID: \"65819aa2-b0f6-48e2-9c17-1a8f35722773\") "
Sep 15 19:21:49.447055 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.446932    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65819aa2-b0f6-48e2-9c17-1a8f35722773-metrics-certs\") pod \"65819aa2-b0f6-48e2-9c17-1a8f35722773\" (UID: \"65819aa2-b0f6-48e2-9c17-1a8f35722773\") "
Sep 15 19:21:49.447233 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.447203    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65819aa2-b0f6-48e2-9c17-1a8f35722773-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "65819aa2-b0f6-48e2-9c17-1a8f35722773" (UID: "65819aa2-b0f6-48e2-9c17-1a8f35722773"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:21:49.449001 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.448975    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65819aa2-b0f6-48e2-9c17-1a8f35722773-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "65819aa2-b0f6-48e2-9c17-1a8f35722773" (UID: "65819aa2-b0f6-48e2-9c17-1a8f35722773"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:21:49.449110 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.449022    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65819aa2-b0f6-48e2-9c17-1a8f35722773-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "65819aa2-b0f6-48e2-9c17-1a8f35722773" (UID: "65819aa2-b0f6-48e2-9c17-1a8f35722773"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:21:49.449228 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.449206    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65819aa2-b0f6-48e2-9c17-1a8f35722773-kube-api-access-nnpdg" (OuterVolumeSpecName: "kube-api-access-nnpdg") pod "65819aa2-b0f6-48e2-9c17-1a8f35722773" (UID: "65819aa2-b0f6-48e2-9c17-1a8f35722773"). InnerVolumeSpecName "kube-api-access-nnpdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:49.548282 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.548248    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65819aa2-b0f6-48e2-9c17-1a8f35722773-maas-api-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:21:49.548282 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.548275    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65819aa2-b0f6-48e2-9c17-1a8f35722773-metrics-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:21:49.548282 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.548284    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnpdg\" (UniqueName: \"kubernetes.io/projected/65819aa2-b0f6-48e2-9c17-1a8f35722773-kube-api-access-nnpdg\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:21:49.548282 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.548294    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65819aa2-b0f6-48e2-9c17-1a8f35722773-openshift-service-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:21:49.717915 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.717821    2555 generic.go:356] "Generic (PLEG): container finished" podID="65819aa2-b0f6-48e2-9c17-1a8f35722773" containerID="9af6f09a3f75517f57f9278c523f1c9ba10132e7abae92c6200183474659bd15" exitCode=0
Sep 15 19:21:49.717915 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.717867    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"
Sep 15 19:21:49.717915 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.717906    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh" event={"ID":"65819aa2-b0f6-48e2-9c17-1a8f35722773","Type":"ContainerDied","Data":"9af6f09a3f75517f57f9278c523f1c9ba10132e7abae92c6200183474659bd15"}
Sep 15 19:21:49.718162 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.717963    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh" event={"ID":"65819aa2-b0f6-48e2-9c17-1a8f35722773","Type":"ContainerDied","Data":"a4832f473c589e045ac96b6ea300cfe450f9a634172551e51eb2aa7a025144b5"}
Sep 15 19:21:49.718162 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.717987    2555 scope.go:117] "RemoveContainer" containerID="9af6f09a3f75517f57f9278c523f1c9ba10132e7abae92c6200183474659bd15"
Sep 15 19:21:49.727492 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.727472    2555 scope.go:117] "RemoveContainer" containerID="9af6f09a3f75517f57f9278c523f1c9ba10132e7abae92c6200183474659bd15"
Sep 15 19:21:49.727810 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:21:49.727790    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af6f09a3f75517f57f9278c523f1c9ba10132e7abae92c6200183474659bd15\": container with ID starting with 9af6f09a3f75517f57f9278c523f1c9ba10132e7abae92c6200183474659bd15 not found: ID does not exist" containerID="9af6f09a3f75517f57f9278c523f1c9ba10132e7abae92c6200183474659bd15"
Sep 15 19:21:49.727870 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.727819    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af6f09a3f75517f57f9278c523f1c9ba10132e7abae92c6200183474659bd15"} err="failed to get container status \"9af6f09a3f75517f57f9278c523f1c9ba10132e7abae92c6200183474659bd15\": rpc error: code = NotFound desc = could not find container \"9af6f09a3f75517f57f9278c523f1c9ba10132e7abae92c6200183474659bd15\": container with ID starting with 9af6f09a3f75517f57f9278c523f1c9ba10132e7abae92c6200183474659bd15 not found: ID does not exist"
Sep 15 19:21:49.738506 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.738481    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"]
Sep 15 19:21:49.744053 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:49.744024    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-25b1b3-79b68944d9-5txrh"]
Sep 15 19:21:51.040426 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:21:51.040395    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65819aa2-b0f6-48e2-9c17-1a8f35722773" path="/var/lib/kubelet/pods/65819aa2-b0f6-48e2-9c17-1a8f35722773/volumes"
Sep 15 19:22:06.816749 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:06.816718    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"]
Sep 15 19:22:06.817322 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:06.816979    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h" podUID="33608ba7-ea93-48ff-b4d8-bbe0dab5ac13" containerName="maas-api" containerID="cri-o://bb104cb7cf63546d6f48e96c31191ba6eafb6ac1a555aeb864a17ac012a0f462" gracePeriod=30
Sep 15 19:22:07.065269 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.065243    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"
Sep 15 19:22:07.121297 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.121215    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-metrics-certs\") pod \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\" (UID: \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\") "
Sep 15 19:22:07.121297 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.121278    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whzlv\" (UniqueName: \"kubernetes.io/projected/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-kube-api-access-whzlv\") pod \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\" (UID: \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\") "
Sep 15 19:22:07.121510 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.121381    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-maas-api-tls\") pod \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\" (UID: \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\") "
Sep 15 19:22:07.121510 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.121450    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-openshift-service-ca\") pod \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\" (UID: \"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13\") "
Sep 15 19:22:07.121838 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.121810    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33608ba7-ea93-48ff-b4d8-bbe0dab5ac13" (UID: "33608ba7-ea93-48ff-b4d8-bbe0dab5ac13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:22:07.123419 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.123387    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "33608ba7-ea93-48ff-b4d8-bbe0dab5ac13" (UID: "33608ba7-ea93-48ff-b4d8-bbe0dab5ac13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:22:07.123522 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.123418    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33608ba7-ea93-48ff-b4d8-bbe0dab5ac13" (UID: "33608ba7-ea93-48ff-b4d8-bbe0dab5ac13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:22:07.123522 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.123437    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-kube-api-access-whzlv" (OuterVolumeSpecName: "kube-api-access-whzlv") pod "33608ba7-ea93-48ff-b4d8-bbe0dab5ac13" (UID: "33608ba7-ea93-48ff-b4d8-bbe0dab5ac13"). InnerVolumeSpecName "kube-api-access-whzlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:22:07.222135 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.222098    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-metrics-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:22:07.222135 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.222126    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whzlv\" (UniqueName: \"kubernetes.io/projected/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-kube-api-access-whzlv\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:22:07.222135 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.222137    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-maas-api-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:22:07.222381 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.222146    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13-openshift-service-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:22:07.810056 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.810016    2555 generic.go:356] "Generic (PLEG): container finished" podID="33608ba7-ea93-48ff-b4d8-bbe0dab5ac13" containerID="bb104cb7cf63546d6f48e96c31191ba6eafb6ac1a555aeb864a17ac012a0f462" exitCode=0
Sep 15 19:22:07.810267 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.810059    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"
Sep 15 19:22:07.810267 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.810103    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h" event={"ID":"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13","Type":"ContainerDied","Data":"bb104cb7cf63546d6f48e96c31191ba6eafb6ac1a555aeb864a17ac012a0f462"}
Sep 15 19:22:07.810267 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.810166    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h" event={"ID":"33608ba7-ea93-48ff-b4d8-bbe0dab5ac13","Type":"ContainerDied","Data":"62f87a8da50f006399d60d01c32ddaa491676e4832eb1c482b4bb73155eda07d"}
Sep 15 19:22:07.810267 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.810197    2555 scope.go:117] "RemoveContainer" containerID="bb104cb7cf63546d6f48e96c31191ba6eafb6ac1a555aeb864a17ac012a0f462"
Sep 15 19:22:07.819683 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.819445    2555 scope.go:117] "RemoveContainer" containerID="bb104cb7cf63546d6f48e96c31191ba6eafb6ac1a555aeb864a17ac012a0f462"
Sep 15 19:22:07.819921 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:22:07.819786    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb104cb7cf63546d6f48e96c31191ba6eafb6ac1a555aeb864a17ac012a0f462\": container with ID starting with bb104cb7cf63546d6f48e96c31191ba6eafb6ac1a555aeb864a17ac012a0f462 not found: ID does not exist" containerID="bb104cb7cf63546d6f48e96c31191ba6eafb6ac1a555aeb864a17ac012a0f462"
Sep 15 19:22:07.819921 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.819813    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb104cb7cf63546d6f48e96c31191ba6eafb6ac1a555aeb864a17ac012a0f462"} err="failed to get container status \"bb104cb7cf63546d6f48e96c31191ba6eafb6ac1a555aeb864a17ac012a0f462\": rpc error: code = NotFound desc = could not find container \"bb104cb7cf63546d6f48e96c31191ba6eafb6ac1a555aeb864a17ac012a0f462\": container with ID starting with bb104cb7cf63546d6f48e96c31191ba6eafb6ac1a555aeb864a17ac012a0f462 not found: ID does not exist"
Sep 15 19:22:07.831593 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.831569    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"]
Sep 15 19:22:07.839357 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:07.839337    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7073b8-cfcb684b-vv78h"]
Sep 15 19:22:09.040058 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:09.040027    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33608ba7-ea93-48ff-b4d8-bbe0dab5ac13" path="/var/lib/kubelet/pods/33608ba7-ea93-48ff-b4d8-bbe0dab5ac13/volumes"
Sep 15 19:22:24.455411 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.455380    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"]
Sep 15 19:22:24.455860 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.455582    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd" podUID="563730fe-d2dd-4e64-a872-d4247466886e" containerName="maas-api" containerID="cri-o://fa941bfc790180ae2a1626f1fdad221a5ad38565131956c6841d5ede13b25448" gracePeriod=30
Sep 15 19:22:24.700814 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.700785    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"
Sep 15 19:22:24.779784 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.779752    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/563730fe-d2dd-4e64-a872-d4247466886e-metrics-certs\") pod \"563730fe-d2dd-4e64-a872-d4247466886e\" (UID: \"563730fe-d2dd-4e64-a872-d4247466886e\") "
Sep 15 19:22:24.779979 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.779849    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsgxj\" (UniqueName: \"kubernetes.io/projected/563730fe-d2dd-4e64-a872-d4247466886e-kube-api-access-tsgxj\") pod \"563730fe-d2dd-4e64-a872-d4247466886e\" (UID: \"563730fe-d2dd-4e64-a872-d4247466886e\") "
Sep 15 19:22:24.779979 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.779955    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563730fe-d2dd-4e64-a872-d4247466886e-maas-api-tls\") pod \"563730fe-d2dd-4e64-a872-d4247466886e\" (UID: \"563730fe-d2dd-4e64-a872-d4247466886e\") "
Sep 15 19:22:24.780105 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.780002    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563730fe-d2dd-4e64-a872-d4247466886e-openshift-service-ca\") pod \"563730fe-d2dd-4e64-a872-d4247466886e\" (UID: \"563730fe-d2dd-4e64-a872-d4247466886e\") "
Sep 15 19:22:24.780357 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.780332    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/563730fe-d2dd-4e64-a872-d4247466886e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "563730fe-d2dd-4e64-a872-d4247466886e" (UID: "563730fe-d2dd-4e64-a872-d4247466886e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:22:24.782224 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.782196    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563730fe-d2dd-4e64-a872-d4247466886e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "563730fe-d2dd-4e64-a872-d4247466886e" (UID: "563730fe-d2dd-4e64-a872-d4247466886e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:22:24.782383 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.782244    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563730fe-d2dd-4e64-a872-d4247466886e-kube-api-access-tsgxj" (OuterVolumeSpecName: "kube-api-access-tsgxj") pod "563730fe-d2dd-4e64-a872-d4247466886e" (UID: "563730fe-d2dd-4e64-a872-d4247466886e"). InnerVolumeSpecName "kube-api-access-tsgxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:22:24.782383 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.782329    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563730fe-d2dd-4e64-a872-d4247466886e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "563730fe-d2dd-4e64-a872-d4247466886e" (UID: "563730fe-d2dd-4e64-a872-d4247466886e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:22:24.881221 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.881183    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsgxj\" (UniqueName: \"kubernetes.io/projected/563730fe-d2dd-4e64-a872-d4247466886e-kube-api-access-tsgxj\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:22:24.881221 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.881215    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563730fe-d2dd-4e64-a872-d4247466886e-maas-api-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:22:24.881221 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.881224    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563730fe-d2dd-4e64-a872-d4247466886e-openshift-service-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:22:24.881221 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.881234    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/563730fe-d2dd-4e64-a872-d4247466886e-metrics-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:22:24.901834 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.901804    2555 generic.go:356] "Generic (PLEG): container finished" podID="563730fe-d2dd-4e64-a872-d4247466886e" containerID="fa941bfc790180ae2a1626f1fdad221a5ad38565131956c6841d5ede13b25448" exitCode=0
Sep 15 19:22:24.901970 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.901851    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"
Sep 15 19:22:24.901970 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.901866    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd" event={"ID":"563730fe-d2dd-4e64-a872-d4247466886e","Type":"ContainerDied","Data":"fa941bfc790180ae2a1626f1fdad221a5ad38565131956c6841d5ede13b25448"}
Sep 15 19:22:24.901970 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.901907    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd" event={"ID":"563730fe-d2dd-4e64-a872-d4247466886e","Type":"ContainerDied","Data":"c4feb95dbc39a234fda92f93d4efbd41772f2bda6c34d78beca3ff3a23780900"}
Sep 15 19:22:24.901970 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.901928    2555 scope.go:117] "RemoveContainer" containerID="fa941bfc790180ae2a1626f1fdad221a5ad38565131956c6841d5ede13b25448"
Sep 15 19:22:24.912278 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.912261    2555 scope.go:117] "RemoveContainer" containerID="fa941bfc790180ae2a1626f1fdad221a5ad38565131956c6841d5ede13b25448"
Sep 15 19:22:24.912528 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:22:24.912509    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa941bfc790180ae2a1626f1fdad221a5ad38565131956c6841d5ede13b25448\": container with ID starting with fa941bfc790180ae2a1626f1fdad221a5ad38565131956c6841d5ede13b25448 not found: ID does not exist" containerID="fa941bfc790180ae2a1626f1fdad221a5ad38565131956c6841d5ede13b25448"
Sep 15 19:22:24.912567 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.912536    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa941bfc790180ae2a1626f1fdad221a5ad38565131956c6841d5ede13b25448"} err="failed to get container status \"fa941bfc790180ae2a1626f1fdad221a5ad38565131956c6841d5ede13b25448\": rpc error: code = NotFound desc = could not find container \"fa941bfc790180ae2a1626f1fdad221a5ad38565131956c6841d5ede13b25448\": container with ID starting with fa941bfc790180ae2a1626f1fdad221a5ad38565131956c6841d5ede13b25448 not found: ID does not exist"
Sep 15 19:22:24.923752 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.923729    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"]
Sep 15 19:22:24.927755 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:24.927722    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-33ff04-659977b4bb-k6vmd"]
Sep 15 19:22:25.040251 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:22:25.040167    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="563730fe-d2dd-4e64-a872-d4247466886e" path="/var/lib/kubelet/pods/563730fe-d2dd-4e64-a872-d4247466886e/volumes"
Sep 15 19:25:07.061047 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:25:07.061011    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn"]
Sep 15 19:25:07.061714 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:25:07.061219    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn" podUID="3038ea59-f062-4a17-b5a1-3655609b4ae3" containerName="manager" containerID="cri-o://805fe67b7ab24a5a82d54f4027f2adb3c3e5727b30c90932998973bb3d8ec0b4" gracePeriod=10
Sep 15 19:25:07.693637 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:25:07.693581    2555 generic.go:356] "Generic (PLEG): container finished" podID="3038ea59-f062-4a17-b5a1-3655609b4ae3" containerID="805fe67b7ab24a5a82d54f4027f2adb3c3e5727b30c90932998973bb3d8ec0b4" exitCode=0
Sep 15 19:25:07.693808 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:25:07.693667    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn" event={"ID":"3038ea59-f062-4a17-b5a1-3655609b4ae3","Type":"ContainerDied","Data":"805fe67b7ab24a5a82d54f4027f2adb3c3e5727b30c90932998973bb3d8ec0b4"}
Sep 15 19:25:08.114368 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:25:08.114341    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn"
Sep 15 19:25:08.211147 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:25:08.211120    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dsbb\" (UniqueName: \"kubernetes.io/projected/3038ea59-f062-4a17-b5a1-3655609b4ae3-kube-api-access-6dsbb\") pod \"3038ea59-f062-4a17-b5a1-3655609b4ae3\" (UID: \"3038ea59-f062-4a17-b5a1-3655609b4ae3\") "
Sep 15 19:25:08.211337 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:25:08.211227    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3038ea59-f062-4a17-b5a1-3655609b4ae3-extensions-socket-volume\") pod \"3038ea59-f062-4a17-b5a1-3655609b4ae3\" (UID: \"3038ea59-f062-4a17-b5a1-3655609b4ae3\") "
Sep 15 19:25:08.211623 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:25:08.211585    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3038ea59-f062-4a17-b5a1-3655609b4ae3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3038ea59-f062-4a17-b5a1-3655609b4ae3" (UID: "3038ea59-f062-4a17-b5a1-3655609b4ae3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:25:08.213171 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:25:08.213148    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3038ea59-f062-4a17-b5a1-3655609b4ae3-kube-api-access-6dsbb" (OuterVolumeSpecName: "kube-api-access-6dsbb") pod "3038ea59-f062-4a17-b5a1-3655609b4ae3" (UID: "3038ea59-f062-4a17-b5a1-3655609b4ae3"). InnerVolumeSpecName "kube-api-access-6dsbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:25:08.312467 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:25:08.312418    2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3038ea59-f062-4a17-b5a1-3655609b4ae3-extensions-socket-volume\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:25:08.312467 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:25:08.312460    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dsbb\" (UniqueName: \"kubernetes.io/projected/3038ea59-f062-4a17-b5a1-3655609b4ae3-kube-api-access-6dsbb\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:25:08.700017 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:25:08.699933    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn"
Sep 15 19:25:08.700017 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:25:08.699973    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn" event={"ID":"3038ea59-f062-4a17-b5a1-3655609b4ae3","Type":"ContainerDied","Data":"a3b6741939e685401a904177aed61ffe0d7367cd65d4a6c60dda9e7aaddf22fb"}
Sep 15 19:25:08.700240 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:25:08.700034    2555 scope.go:117] "RemoveContainer" containerID="805fe67b7ab24a5a82d54f4027f2adb3c3e5727b30c90932998973bb3d8ec0b4"
Sep 15 19:25:08.721864 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:25:08.721839    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn"]
Sep 15 19:25:08.726370 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:25:08.726339    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j4chn"]
Sep 15 19:25:09.039486 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:25:09.039455    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3038ea59-f062-4a17-b5a1-3655609b4ae3" path="/var/lib/kubelet/pods/3038ea59-f062-4a17-b5a1-3655609b4ae3/volumes"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.903695    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qzn8z"]
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.904751    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="563730fe-d2dd-4e64-a872-d4247466886e" containerName="maas-api"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.904773    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="563730fe-d2dd-4e64-a872-d4247466886e" containerName="maas-api"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.904801    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3038ea59-f062-4a17-b5a1-3655609b4ae3" containerName="manager"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.904809    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3038ea59-f062-4a17-b5a1-3655609b4ae3" containerName="manager"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.904844    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33608ba7-ea93-48ff-b4d8-bbe0dab5ac13" containerName="maas-api"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.904855    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="33608ba7-ea93-48ff-b4d8-bbe0dab5ac13" containerName="maas-api"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.904867    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a23682a5-793e-4b1e-838f-f812f337ac40" containerName="maas-api"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.904875    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a23682a5-793e-4b1e-838f-f812f337ac40" containerName="maas-api"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.904884    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65819aa2-b0f6-48e2-9c17-1a8f35722773" containerName="maas-api"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.904892    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="65819aa2-b0f6-48e2-9c17-1a8f35722773" containerName="maas-api"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.904912    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc57f241-7447-4f92-874b-df716f1e5b05" containerName="maas-api"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.904920    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc57f241-7447-4f92-874b-df716f1e5b05" containerName="maas-api"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.904944    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4edb8dc8-4604-41c3-bca6-2c38fa97c6dd" containerName="main"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.904952    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4edb8dc8-4604-41c3-bca6-2c38fa97c6dd" containerName="main"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.905125    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3038ea59-f062-4a17-b5a1-3655609b4ae3" containerName="manager"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.905142    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="65819aa2-b0f6-48e2-9c17-1a8f35722773" containerName="maas-api"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.905154    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc57f241-7447-4f92-874b-df716f1e5b05" containerName="maas-api"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.905168    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="33608ba7-ea93-48ff-b4d8-bbe0dab5ac13" containerName="maas-api"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.905178    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="563730fe-d2dd-4e64-a872-d4247466886e" containerName="maas-api"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.905187    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a23682a5-793e-4b1e-838f-f812f337ac40" containerName="maas-api"
Sep 15 19:26:07.907061 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.905202    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4edb8dc8-4604-41c3-bca6-2c38fa97c6dd" containerName="main"
Sep 15 19:26:07.909866 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.909299    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qzn8z"
Sep 15 19:26:07.912056 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.912034    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lhsr2\""
Sep 15 19:26:07.918836 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:07.918810    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qzn8z"]
Sep 15 19:26:08.078264 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:08.078224    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3df758a-73f5-47dc-8c0d-a48bd65db15b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-qzn8z\" (UID: \"c3df758a-73f5-47dc-8c0d-a48bd65db15b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qzn8z"
Sep 15 19:26:08.078442 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:08.078271    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfpvg\" (UniqueName: \"kubernetes.io/projected/c3df758a-73f5-47dc-8c0d-a48bd65db15b-kube-api-access-kfpvg\") pod \"kuadrant-operator-controller-manager-5769f99d9f-qzn8z\" (UID: \"c3df758a-73f5-47dc-8c0d-a48bd65db15b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qzn8z"
Sep 15 19:26:08.179637 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:08.179541    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3df758a-73f5-47dc-8c0d-a48bd65db15b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-qzn8z\" (UID: \"c3df758a-73f5-47dc-8c0d-a48bd65db15b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qzn8z"
Sep 15 19:26:08.179637 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:08.179577    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfpvg\" (UniqueName: \"kubernetes.io/projected/c3df758a-73f5-47dc-8c0d-a48bd65db15b-kube-api-access-kfpvg\") pod \"kuadrant-operator-controller-manager-5769f99d9f-qzn8z\" (UID: \"c3df758a-73f5-47dc-8c0d-a48bd65db15b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qzn8z"
Sep 15 19:26:08.179938 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:08.179916    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3df758a-73f5-47dc-8c0d-a48bd65db15b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-qzn8z\" (UID: \"c3df758a-73f5-47dc-8c0d-a48bd65db15b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qzn8z"
Sep 15 19:26:08.190206 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:08.190179    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfpvg\" (UniqueName: \"kubernetes.io/projected/c3df758a-73f5-47dc-8c0d-a48bd65db15b-kube-api-access-kfpvg\") pod \"kuadrant-operator-controller-manager-5769f99d9f-qzn8z\" (UID: \"c3df758a-73f5-47dc-8c0d-a48bd65db15b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qzn8z"
Sep 15 19:26:08.220953 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:08.220920    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qzn8z"
Sep 15 19:26:08.514503 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:08.514465    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qzn8z"]
Sep 15 19:26:08.519803 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:26:08.519771    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3df758a_73f5_47dc_8c0d_a48bd65db15b.slice/crio-2068583076addda3040be8c8241caf51808060c64c8cedf16548eb0a3a8463b0 WatchSource:0}: Error finding container 2068583076addda3040be8c8241caf51808060c64c8cedf16548eb0a3a8463b0: Status 404 returned error can't find the container with id 2068583076addda3040be8c8241caf51808060c64c8cedf16548eb0a3a8463b0
Sep 15 19:26:08.988305 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:08.988216    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qzn8z" event={"ID":"c3df758a-73f5-47dc-8c0d-a48bd65db15b","Type":"ContainerStarted","Data":"3cc85f49304f0e81e50125200a95279420d7838f47ab169126fbec8b8a7a15e4"}
Sep 15 19:26:08.988305 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:08.988268    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qzn8z" event={"ID":"c3df758a-73f5-47dc-8c0d-a48bd65db15b","Type":"ContainerStarted","Data":"2068583076addda3040be8c8241caf51808060c64c8cedf16548eb0a3a8463b0"}
Sep 15 19:26:08.988728 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:08.988350    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qzn8z"
Sep 15 19:26:09.018567 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:09.018517    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qzn8z" podStartSLOduration=2.018497643 podStartE2EDuration="2.018497643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:26:09.014355217 +0000 UTC m=+2110.501684202" watchObservedRunningTime="2026-09-15 19:26:09.018497643 +0000 UTC m=+2110.505826631"
Sep 15 19:26:19.995586 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:19.995552    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qzn8z"
Sep 15 19:26:52.339559 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.339483    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"]
Sep 15 19:26:52.342835 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.342820    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.345169 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.345143    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-6076da03-kserve-self-signed-certs\""
Sep 15 19:26:52.350833 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.350809    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"]
Sep 15 19:26:52.392896 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.392868    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-model-cache\") pod \"e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") " pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.392896 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.392900    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e33755-2842-47ba-88b9-92d4f52ac679-tls-certs\") pod \"e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") " pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.393084 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.392933    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhw4x\" (UniqueName: \"kubernetes.io/projected/03e33755-2842-47ba-88b9-92d4f52ac679-kube-api-access-mhw4x\") pod \"e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") " pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.393084 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.392950    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-dshm\") pod \"e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") " pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.393084 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.393054    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-tmp-dir\") pod \"e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") " pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.393084 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.393080    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-home\") pod \"e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") " pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.493549 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.493510    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-home\") pod \"e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") " pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.493754 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.493560    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-model-cache\") pod \"e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") " pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.493754 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.493584    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e33755-2842-47ba-88b9-92d4f52ac679-tls-certs\") pod \"e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") " pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.493754 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.493635    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhw4x\" (UniqueName: \"kubernetes.io/projected/03e33755-2842-47ba-88b9-92d4f52ac679-kube-api-access-mhw4x\") pod \"e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") " pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.493754 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.493663    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-dshm\") pod \"e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") " pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.493982 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.493824    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-tmp-dir\") pod \"e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") " pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.493982 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.493936    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-home\") pod \"e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") " pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.494073 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.493996    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-model-cache\") pod \"e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") " pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.494124 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.494107    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-tmp-dir\") pod \"e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") " pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.495923 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.495901    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-dshm\") pod \"e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") " pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.496092 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.496075    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e33755-2842-47ba-88b9-92d4f52ac679-tls-certs\") pod \"e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") " pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.500894 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.500876    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhw4x\" (UniqueName: \"kubernetes.io/projected/03e33755-2842-47ba-88b9-92d4f52ac679-kube-api-access-mhw4x\") pod \"e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") " pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.653083 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.652991    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:26:52.925089 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:52.925061    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"]
Sep 15 19:26:52.926875 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:26:52.926849    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03e33755_2842_47ba_88b9_92d4f52ac679.slice/crio-3090d6126e01973f1b322da5664d11357d50bbc8a8de0b42c7e36d2d1e886836 WatchSource:0}: Error finding container 3090d6126e01973f1b322da5664d11357d50bbc8a8de0b42c7e36d2d1e886836: Status 404 returned error can't find the container with id 3090d6126e01973f1b322da5664d11357d50bbc8a8de0b42c7e36d2d1e886836
Sep 15 19:26:53.205451 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:53.205361    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm" event={"ID":"03e33755-2842-47ba-88b9-92d4f52ac679","Type":"ContainerStarted","Data":"2353ac026853e841edef2c639b60f1fb22c722bad68d457a9267897b13f5e720"}
Sep 15 19:26:53.205451 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:53.205406    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm" event={"ID":"03e33755-2842-47ba-88b9-92d4f52ac679","Type":"ContainerStarted","Data":"3090d6126e01973f1b322da5664d11357d50bbc8a8de0b42c7e36d2d1e886836"}
Sep 15 19:26:53.222323 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:26:53.222274    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm" podStartSLOduration=1.222257586 podStartE2EDuration="1.222257586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:26:53.220824607 +0000 UTC m=+2154.708153592" watchObservedRunningTime="2026-09-15 19:26:53.222257586 +0000 UTC m=+2154.709586570"
Sep 15 19:27:02.653362 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:02.653323    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:27:02.653362 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:02.653369    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:27:02.665815 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:02.665790    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:27:03.265528 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:03.265489    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:27:08.638020 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.637985    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"]
Sep 15 19:27:08.641808 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.641780    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:08.644400 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.644373    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-6076da03-kserve-self-signed-certs\""
Sep 15 19:27:08.652977 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.652944    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"]
Sep 15 19:27:08.752330 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.752285    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmfqx\" (UniqueName: \"kubernetes.io/projected/53a40583-d63e-4497-be74-9df7642492d2-kube-api-access-pmfqx\") pod \"e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") " pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:08.752540 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.752343    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53a40583-d63e-4497-be74-9df7642492d2-tls-certs\") pod \"e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") " pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:08.752540 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.752374    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-dshm\") pod \"e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") " pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:08.752540 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.752417    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-tmp-dir\") pod \"e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") " pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:08.752540 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.752460    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-model-cache\") pod \"e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") " pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:08.752540 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.752492    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-home\") pod \"e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") " pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:08.853452 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.853413    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmfqx\" (UniqueName: \"kubernetes.io/projected/53a40583-d63e-4497-be74-9df7642492d2-kube-api-access-pmfqx\") pod \"e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") " pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:08.853452 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.853459    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53a40583-d63e-4497-be74-9df7642492d2-tls-certs\") pod \"e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") " pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:08.853778 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.853478    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-dshm\") pod \"e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") " pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:08.853778 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.853509    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-tmp-dir\") pod \"e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") " pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:08.853778 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.853528    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-model-cache\") pod \"e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") " pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:08.853778 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.853549    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-home\") pod \"e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") " pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:08.854044 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.854022    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-tmp-dir\") pod \"e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") " pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:08.854106 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.854045    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-model-cache\") pod \"e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") " pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:08.854106 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.854061    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-home\") pod \"e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") " pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:08.855933 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.855912    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-dshm\") pod \"e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") " pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:08.856054 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.856040    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53a40583-d63e-4497-be74-9df7642492d2-tls-certs\") pod \"e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") " pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:08.861940 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.861913    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmfqx\" (UniqueName: \"kubernetes.io/projected/53a40583-d63e-4497-be74-9df7642492d2-kube-api-access-pmfqx\") pod \"e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") " pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:08.958070 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:08.957966    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:09.229519 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:09.229494    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"]
Sep 15 19:27:09.231370 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:27:09.231328    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53a40583_d63e_4497_be74_9df7642492d2.slice/crio-3ba942f3fb403feccb34ec93eddffbfb5886754f7c8c9cc7122fd902d833d585 WatchSource:0}: Error finding container 3ba942f3fb403feccb34ec93eddffbfb5886754f7c8c9cc7122fd902d833d585: Status 404 returned error can't find the container with id 3ba942f3fb403feccb34ec93eddffbfb5886754f7c8c9cc7122fd902d833d585
Sep 15 19:27:09.285902 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:09.285861    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j" event={"ID":"53a40583-d63e-4497-be74-9df7642492d2","Type":"ContainerStarted","Data":"3ba942f3fb403feccb34ec93eddffbfb5886754f7c8c9cc7122fd902d833d585"}
Sep 15 19:27:10.293375 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:10.293338    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j" event={"ID":"53a40583-d63e-4497-be74-9df7642492d2","Type":"ContainerStarted","Data":"5e1013d7076a283e9a7d9f8e5da0272ce670ae89d5c8672eb7c5f09246a72993"}
Sep 15 19:27:10.308888 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:10.308830    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j" podStartSLOduration=2.308810934 podStartE2EDuration="2.308810934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:27:10.308426021 +0000 UTC m=+2171.795755006" watchObservedRunningTime="2026-09-15 19:27:10.308810934 +0000 UTC m=+2171.796139920"
Sep 15 19:27:18.958590 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:18.958552    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:18.958590 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:18.958599    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:18.971048 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:18.971022    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:19.354136 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:19.354107    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:23.346286 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:23.346253    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"]
Sep 15 19:27:24.205534 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:24.205499    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"]
Sep 15 19:27:38.367373 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.367327    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm" podUID="03e33755-2842-47ba-88b9-92d4f52ac679" containerName="main" containerID="cri-o://2353ac026853e841edef2c639b60f1fb22c722bad68d457a9267897b13f5e720" gracePeriod=45
Sep 15 19:27:38.610617 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.610591    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:27:38.638355 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.638286    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-home\") pod \"03e33755-2842-47ba-88b9-92d4f52ac679\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") "
Sep 15 19:27:38.638355 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.638329    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-dshm\") pod \"03e33755-2842-47ba-88b9-92d4f52ac679\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") "
Sep 15 19:27:38.638355 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.638346    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-model-cache\") pod \"03e33755-2842-47ba-88b9-92d4f52ac679\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") "
Sep 15 19:27:38.638633 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.638367    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhw4x\" (UniqueName: \"kubernetes.io/projected/03e33755-2842-47ba-88b9-92d4f52ac679-kube-api-access-mhw4x\") pod \"03e33755-2842-47ba-88b9-92d4f52ac679\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") "
Sep 15 19:27:38.638633 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.638482    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-tmp-dir\") pod \"03e33755-2842-47ba-88b9-92d4f52ac679\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") "
Sep 15 19:27:38.638633 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.638546    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e33755-2842-47ba-88b9-92d4f52ac679-tls-certs\") pod \"03e33755-2842-47ba-88b9-92d4f52ac679\" (UID: \"03e33755-2842-47ba-88b9-92d4f52ac679\") "
Sep 15 19:27:38.638633 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.638580    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-home" (OuterVolumeSpecName: "home") pod "03e33755-2842-47ba-88b9-92d4f52ac679" (UID: "03e33755-2842-47ba-88b9-92d4f52ac679"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:27:38.638633 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.638591    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-model-cache" (OuterVolumeSpecName: "model-cache") pod "03e33755-2842-47ba-88b9-92d4f52ac679" (UID: "03e33755-2842-47ba-88b9-92d4f52ac679"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:27:38.638912 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.638794    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "03e33755-2842-47ba-88b9-92d4f52ac679" (UID: "03e33755-2842-47ba-88b9-92d4f52ac679"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:27:38.638912 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.638895    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-tmp-dir\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:27:38.638912 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.638914    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-home\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:27:38.639044 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.638925    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-model-cache\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:27:38.640578 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.640551    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-dshm" (OuterVolumeSpecName: "dshm") pod "03e33755-2842-47ba-88b9-92d4f52ac679" (UID: "03e33755-2842-47ba-88b9-92d4f52ac679"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:27:38.640949 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.640929    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e33755-2842-47ba-88b9-92d4f52ac679-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03e33755-2842-47ba-88b9-92d4f52ac679" (UID: "03e33755-2842-47ba-88b9-92d4f52ac679"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:27:38.641072 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.641055    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e33755-2842-47ba-88b9-92d4f52ac679-kube-api-access-mhw4x" (OuterVolumeSpecName: "kube-api-access-mhw4x") pod "03e33755-2842-47ba-88b9-92d4f52ac679" (UID: "03e33755-2842-47ba-88b9-92d4f52ac679"). InnerVolumeSpecName "kube-api-access-mhw4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:27:38.740387 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.740352    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e33755-2842-47ba-88b9-92d4f52ac679-dshm\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:27:38.740387 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.740382    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhw4x\" (UniqueName: \"kubernetes.io/projected/03e33755-2842-47ba-88b9-92d4f52ac679-kube-api-access-mhw4x\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:27:38.740387 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:38.740392    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e33755-2842-47ba-88b9-92d4f52ac679-tls-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:27:39.225562 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.225526    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j" podUID="53a40583-d63e-4497-be74-9df7642492d2" containerName="main" containerID="cri-o://5e1013d7076a283e9a7d9f8e5da0272ce670ae89d5c8672eb7c5f09246a72993" gracePeriod=45
Sep 15 19:27:39.343574 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.343547    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j" podUID="53a40583-d63e-4497-be74-9df7642492d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 15 19:27:39.443994 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.443964    2555 generic.go:356] "Generic (PLEG): container finished" podID="53a40583-d63e-4497-be74-9df7642492d2" containerID="5e1013d7076a283e9a7d9f8e5da0272ce670ae89d5c8672eb7c5f09246a72993" exitCode=0
Sep 15 19:27:39.444411 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.444037    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j" event={"ID":"53a40583-d63e-4497-be74-9df7642492d2","Type":"ContainerDied","Data":"5e1013d7076a283e9a7d9f8e5da0272ce670ae89d5c8672eb7c5f09246a72993"}
Sep 15 19:27:39.445504 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.445479    2555 generic.go:356] "Generic (PLEG): container finished" podID="03e33755-2842-47ba-88b9-92d4f52ac679" containerID="2353ac026853e841edef2c639b60f1fb22c722bad68d457a9267897b13f5e720" exitCode=0
Sep 15 19:27:39.445640 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.445531    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm" event={"ID":"03e33755-2842-47ba-88b9-92d4f52ac679","Type":"ContainerDied","Data":"2353ac026853e841edef2c639b60f1fb22c722bad68d457a9267897b13f5e720"}
Sep 15 19:27:39.445640 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.445543    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"
Sep 15 19:27:39.445640 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.445563    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm" event={"ID":"03e33755-2842-47ba-88b9-92d4f52ac679","Type":"ContainerDied","Data":"3090d6126e01973f1b322da5664d11357d50bbc8a8de0b42c7e36d2d1e886836"}
Sep 15 19:27:39.445640 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.445581    2555 scope.go:117] "RemoveContainer" containerID="2353ac026853e841edef2c639b60f1fb22c722bad68d457a9267897b13f5e720"
Sep 15 19:27:39.467088 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.467060    2555 scope.go:117] "RemoveContainer" containerID="2353ac026853e841edef2c639b60f1fb22c722bad68d457a9267897b13f5e720"
Sep 15 19:27:39.467338 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:27:39.467316    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2353ac026853e841edef2c639b60f1fb22c722bad68d457a9267897b13f5e720\": container with ID starting with 2353ac026853e841edef2c639b60f1fb22c722bad68d457a9267897b13f5e720 not found: ID does not exist" containerID="2353ac026853e841edef2c639b60f1fb22c722bad68d457a9267897b13f5e720"
Sep 15 19:27:39.467406 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.467350    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2353ac026853e841edef2c639b60f1fb22c722bad68d457a9267897b13f5e720"} err="failed to get container status \"2353ac026853e841edef2c639b60f1fb22c722bad68d457a9267897b13f5e720\": rpc error: code = NotFound desc = could not find container \"2353ac026853e841edef2c639b60f1fb22c722bad68d457a9267897b13f5e720\": container with ID starting with 2353ac026853e841edef2c639b60f1fb22c722bad68d457a9267897b13f5e720 not found: ID does not exist"
Sep 15 19:27:39.470486 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.470470    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:39.480191 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.480168    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"]
Sep 15 19:27:39.483817 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.483772    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-6076da03-kserve-7666988c59-cnzfm"]
Sep 15 19:27:39.548055 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.548026    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53a40583-d63e-4497-be74-9df7642492d2-tls-certs\") pod \"53a40583-d63e-4497-be74-9df7642492d2\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") "
Sep 15 19:27:39.548217 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.548108    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-dshm\") pod \"53a40583-d63e-4497-be74-9df7642492d2\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") "
Sep 15 19:27:39.548217 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.548164    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-tmp-dir\") pod \"53a40583-d63e-4497-be74-9df7642492d2\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") "
Sep 15 19:27:39.548217 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.548182    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-home\") pod \"53a40583-d63e-4497-be74-9df7642492d2\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") "
Sep 15 19:27:39.548217 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.548208    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmfqx\" (UniqueName: \"kubernetes.io/projected/53a40583-d63e-4497-be74-9df7642492d2-kube-api-access-pmfqx\") pod \"53a40583-d63e-4497-be74-9df7642492d2\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") "
Sep 15 19:27:39.548426 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.548253    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-model-cache\") pod \"53a40583-d63e-4497-be74-9df7642492d2\" (UID: \"53a40583-d63e-4497-be74-9df7642492d2\") "
Sep 15 19:27:39.548486 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.548416    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53a40583-d63e-4497-be74-9df7642492d2" (UID: "53a40583-d63e-4497-be74-9df7642492d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:27:39.548543 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.548507    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-home" (OuterVolumeSpecName: "home") pod "53a40583-d63e-4497-be74-9df7642492d2" (UID: "53a40583-d63e-4497-be74-9df7642492d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:27:39.548598 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.548537    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "53a40583-d63e-4497-be74-9df7642492d2" (UID: "53a40583-d63e-4497-be74-9df7642492d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:27:39.550193 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.550170    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-dshm" (OuterVolumeSpecName: "dshm") pod "53a40583-d63e-4497-be74-9df7642492d2" (UID: "53a40583-d63e-4497-be74-9df7642492d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:27:39.550271 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.550254    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53a40583-d63e-4497-be74-9df7642492d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53a40583-d63e-4497-be74-9df7642492d2" (UID: "53a40583-d63e-4497-be74-9df7642492d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:27:39.550314 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.550283    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a40583-d63e-4497-be74-9df7642492d2-kube-api-access-pmfqx" (OuterVolumeSpecName: "kube-api-access-pmfqx") pod "53a40583-d63e-4497-be74-9df7642492d2" (UID: "53a40583-d63e-4497-be74-9df7642492d2"). InnerVolumeSpecName "kube-api-access-pmfqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:27:39.650032 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.649948    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-tmp-dir\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:27:39.650032 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.649973    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-home\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:27:39.650032 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.649983    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmfqx\" (UniqueName: \"kubernetes.io/projected/53a40583-d63e-4497-be74-9df7642492d2-kube-api-access-pmfqx\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:27:39.650032 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.649995    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-model-cache\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:27:39.650032 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.650003    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53a40583-d63e-4497-be74-9df7642492d2-tls-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:27:39.650032 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:39.650010    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53a40583-d63e-4497-be74-9df7642492d2-dshm\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:27:40.452759 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:40.452732    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"
Sep 15 19:27:40.453226 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:40.452762    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j" event={"ID":"53a40583-d63e-4497-be74-9df7642492d2","Type":"ContainerDied","Data":"3ba942f3fb403feccb34ec93eddffbfb5886754f7c8c9cc7122fd902d833d585"}
Sep 15 19:27:40.453226 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:40.452813    2555 scope.go:117] "RemoveContainer" containerID="5e1013d7076a283e9a7d9f8e5da0272ce670ae89d5c8672eb7c5f09246a72993"
Sep 15 19:27:40.474945 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:40.474916    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"]
Sep 15 19:27:40.479373 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:40.479347    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-6076da03-kserve-658cfb9ccf-btp4j"]
Sep 15 19:27:41.039737 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.039705    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03e33755-2842-47ba-88b9-92d4f52ac679" path="/var/lib/kubelet/pods/03e33755-2842-47ba-88b9-92d4f52ac679/volumes"
Sep 15 19:27:41.040124 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.040112    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53a40583-d63e-4497-be74-9df7642492d2" path="/var/lib/kubelet/pods/53a40583-d63e-4497-be74-9df7642492d2/volumes"
Sep 15 19:27:41.735168 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.735130    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"]
Sep 15 19:27:41.735546 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.735514    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03e33755-2842-47ba-88b9-92d4f52ac679" containerName="main"
Sep 15 19:27:41.735546 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.735525    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="03e33755-2842-47ba-88b9-92d4f52ac679" containerName="main"
Sep 15 19:27:41.735546 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.735533    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53a40583-d63e-4497-be74-9df7642492d2" containerName="main"
Sep 15 19:27:41.735546 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.735538    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="53a40583-d63e-4497-be74-9df7642492d2" containerName="main"
Sep 15 19:27:41.735733 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.735620    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="03e33755-2842-47ba-88b9-92d4f52ac679" containerName="main"
Sep 15 19:27:41.735733 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.735632    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="53a40583-d63e-4497-be74-9df7642492d2" containerName="main"
Sep 15 19:27:41.739884 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.739862    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"
Sep 15 19:27:41.743416 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.743389    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5ksbj\""
Sep 15 19:27:41.743416 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.743409    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:27:41.744221 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.744205    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 15 19:27:41.744330 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.744313    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 19:27:41.744412 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.744396    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 19:27:41.754019 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.753994    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m"]
Sep 15 19:27:41.758096 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.758072    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"]
Sep 15 19:27:41.758223 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.758191    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m"
Sep 15 19:27:41.769928 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.769901    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m"]
Sep 15 19:27:41.869734 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.869699    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/553090b3-775d-46a5-9b36-efaf6a2e81df-maas-api-tls\") pod \"maas-api-59f6566bbc-slqhh\" (UID: \"553090b3-775d-46a5-9b36-efaf6a2e81df\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"
Sep 15 19:27:41.869891 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.869764    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca22ba2-2b93-4d32-81ee-ca6a6de62985-openshift-service-ca\") pod \"maas-api-86d9ddc875-dtb2m\" (UID: \"5ca22ba2-2b93-4d32-81ee-ca6a6de62985\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m"
Sep 15 19:27:41.869891 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.869830    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ca22ba2-2b93-4d32-81ee-ca6a6de62985-maas-api-tls\") pod \"maas-api-86d9ddc875-dtb2m\" (UID: \"5ca22ba2-2b93-4d32-81ee-ca6a6de62985\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m"
Sep 15 19:27:41.869891 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.869868    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ca22ba2-2b93-4d32-81ee-ca6a6de62985-metrics-certs\") pod \"maas-api-86d9ddc875-dtb2m\" (UID: \"5ca22ba2-2b93-4d32-81ee-ca6a6de62985\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m"
Sep 15 19:27:41.869891 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.869885    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/553090b3-775d-46a5-9b36-efaf6a2e81df-metrics-certs\") pod \"maas-api-59f6566bbc-slqhh\" (UID: \"553090b3-775d-46a5-9b36-efaf6a2e81df\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"
Sep 15 19:27:41.870101 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.869975    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/553090b3-775d-46a5-9b36-efaf6a2e81df-openshift-service-ca\") pod \"maas-api-59f6566bbc-slqhh\" (UID: \"553090b3-775d-46a5-9b36-efaf6a2e81df\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"
Sep 15 19:27:41.870101 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.870000    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hmv4\" (UniqueName: \"kubernetes.io/projected/553090b3-775d-46a5-9b36-efaf6a2e81df-kube-api-access-6hmv4\") pod \"maas-api-59f6566bbc-slqhh\" (UID: \"553090b3-775d-46a5-9b36-efaf6a2e81df\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"
Sep 15 19:27:41.870101 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.870017    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmc5q\" (UniqueName: \"kubernetes.io/projected/5ca22ba2-2b93-4d32-81ee-ca6a6de62985-kube-api-access-wmc5q\") pod \"maas-api-86d9ddc875-dtb2m\" (UID: \"5ca22ba2-2b93-4d32-81ee-ca6a6de62985\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m"
Sep 15 19:27:41.971224 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.971180    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/553090b3-775d-46a5-9b36-efaf6a2e81df-openshift-service-ca\") pod \"maas-api-59f6566bbc-slqhh\" (UID: \"553090b3-775d-46a5-9b36-efaf6a2e81df\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"
Sep 15 19:27:41.971224 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.971226    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmv4\" (UniqueName: \"kubernetes.io/projected/553090b3-775d-46a5-9b36-efaf6a2e81df-kube-api-access-6hmv4\") pod \"maas-api-59f6566bbc-slqhh\" (UID: \"553090b3-775d-46a5-9b36-efaf6a2e81df\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"
Sep 15 19:27:41.971523 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.971245    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmc5q\" (UniqueName: \"kubernetes.io/projected/5ca22ba2-2b93-4d32-81ee-ca6a6de62985-kube-api-access-wmc5q\") pod \"maas-api-86d9ddc875-dtb2m\" (UID: \"5ca22ba2-2b93-4d32-81ee-ca6a6de62985\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m"
Sep 15 19:27:41.971523 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.971285    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/553090b3-775d-46a5-9b36-efaf6a2e81df-maas-api-tls\") pod \"maas-api-59f6566bbc-slqhh\" (UID: \"553090b3-775d-46a5-9b36-efaf6a2e81df\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"
Sep 15 19:27:41.971523 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.971336    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca22ba2-2b93-4d32-81ee-ca6a6de62985-openshift-service-ca\") pod \"maas-api-86d9ddc875-dtb2m\" (UID: \"5ca22ba2-2b93-4d32-81ee-ca6a6de62985\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m"
Sep 15 19:27:41.971523 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.971365    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ca22ba2-2b93-4d32-81ee-ca6a6de62985-maas-api-tls\") pod \"maas-api-86d9ddc875-dtb2m\" (UID: \"5ca22ba2-2b93-4d32-81ee-ca6a6de62985\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m"
Sep 15 19:27:41.971785 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.971520    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ca22ba2-2b93-4d32-81ee-ca6a6de62985-metrics-certs\") pod \"maas-api-86d9ddc875-dtb2m\" (UID: \"5ca22ba2-2b93-4d32-81ee-ca6a6de62985\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m"
Sep 15 19:27:41.971785 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.971560    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/553090b3-775d-46a5-9b36-efaf6a2e81df-metrics-certs\") pod \"maas-api-59f6566bbc-slqhh\" (UID: \"553090b3-775d-46a5-9b36-efaf6a2e81df\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"
Sep 15 19:27:41.972082 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.972059    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/553090b3-775d-46a5-9b36-efaf6a2e81df-openshift-service-ca\") pod \"maas-api-59f6566bbc-slqhh\" (UID: \"553090b3-775d-46a5-9b36-efaf6a2e81df\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"
Sep 15 19:27:41.972195 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.972180    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca22ba2-2b93-4d32-81ee-ca6a6de62985-openshift-service-ca\") pod \"maas-api-86d9ddc875-dtb2m\" (UID: \"5ca22ba2-2b93-4d32-81ee-ca6a6de62985\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m"
Sep 15 19:27:41.974125 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.974101    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/553090b3-775d-46a5-9b36-efaf6a2e81df-metrics-certs\") pod \"maas-api-59f6566bbc-slqhh\" (UID: \"553090b3-775d-46a5-9b36-efaf6a2e81df\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"
Sep 15 19:27:41.974204 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.974153    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ca22ba2-2b93-4d32-81ee-ca6a6de62985-metrics-certs\") pod \"maas-api-86d9ddc875-dtb2m\" (UID: \"5ca22ba2-2b93-4d32-81ee-ca6a6de62985\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m"
Sep 15 19:27:41.974317 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.974301    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ca22ba2-2b93-4d32-81ee-ca6a6de62985-maas-api-tls\") pod \"maas-api-86d9ddc875-dtb2m\" (UID: \"5ca22ba2-2b93-4d32-81ee-ca6a6de62985\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m"
Sep 15 19:27:41.974355 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.974311    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/553090b3-775d-46a5-9b36-efaf6a2e81df-maas-api-tls\") pod \"maas-api-59f6566bbc-slqhh\" (UID: \"553090b3-775d-46a5-9b36-efaf6a2e81df\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"
Sep 15 19:27:41.978539 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.978517    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmc5q\" (UniqueName: \"kubernetes.io/projected/5ca22ba2-2b93-4d32-81ee-ca6a6de62985-kube-api-access-wmc5q\") pod \"maas-api-86d9ddc875-dtb2m\" (UID: \"5ca22ba2-2b93-4d32-81ee-ca6a6de62985\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m"
Sep 15 19:27:41.978653 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:41.978555    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmv4\" (UniqueName: \"kubernetes.io/projected/553090b3-775d-46a5-9b36-efaf6a2e81df-kube-api-access-6hmv4\") pod \"maas-api-59f6566bbc-slqhh\" (UID: \"553090b3-775d-46a5-9b36-efaf6a2e81df\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"
Sep 15 19:27:42.050330 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:42.050302    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"
Sep 15 19:27:42.071318 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:42.071289    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m"
Sep 15 19:27:42.327161 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:42.327084    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"]
Sep 15 19:27:42.330514 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:27:42.330489    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553090b3_775d_46a5_9b36_efaf6a2e81df.slice/crio-80638ccd5902d6e60ca64025bd5abac85bcd6e5564fe6047975e7f49c07413fc WatchSource:0}: Error finding container 80638ccd5902d6e60ca64025bd5abac85bcd6e5564fe6047975e7f49c07413fc: Status 404 returned error can't find the container with id 80638ccd5902d6e60ca64025bd5abac85bcd6e5564fe6047975e7f49c07413fc
Sep 15 19:27:42.331891 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:42.331869    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:27:42.356944 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:42.355795    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m"]
Sep 15 19:27:42.466440 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:42.466393    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh" event={"ID":"553090b3-775d-46a5-9b36-efaf6a2e81df","Type":"ContainerStarted","Data":"80638ccd5902d6e60ca64025bd5abac85bcd6e5564fe6047975e7f49c07413fc"}
Sep 15 19:27:42.467645 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:42.467599    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m" event={"ID":"5ca22ba2-2b93-4d32-81ee-ca6a6de62985","Type":"ContainerStarted","Data":"78b97e85ac320f343a59b776ec80d23ef69a70aae4d001c0d1d64e62db4f781d"}
Sep 15 19:27:43.474464 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:43.474409    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m" event={"ID":"5ca22ba2-2b93-4d32-81ee-ca6a6de62985","Type":"ContainerStarted","Data":"ae683dc8b76e028f037c35e09dca82bca9010ce86c700c90ae0c3392fc3ab2f5"}
Sep 15 19:27:43.474969 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:43.474493    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m"
Sep 15 19:27:43.476012 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:43.475988    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh" event={"ID":"553090b3-775d-46a5-9b36-efaf6a2e81df","Type":"ContainerStarted","Data":"d5d53d1618f8291328ecaa1f674edd44e9c47b19b01760ea1b2050ea7e7902d3"}
Sep 15 19:27:43.476110 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:43.476033    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"
Sep 15 19:27:43.492905 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:43.492861    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m" podStartSLOduration=2.156788854 podStartE2EDuration="2.492846812s" totalImagesPullingTime="336.057958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:27:43.491235638 +0000 UTC m=+2204.978564623" watchObservedRunningTime="2026-09-15 19:27:43.492846812 +0000 UTC m=+2204.980175796"
Sep 15 19:27:43.506328 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:43.506280    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh" podStartSLOduration=2.173873096 podStartE2EDuration="2.506263021s" totalImagesPullingTime="332.389925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:27:43.505923873 +0000 UTC m=+2204.993252858" watchObservedRunningTime="2026-09-15 19:27:43.506263021 +0000 UTC m=+2204.993592007"
Sep 15 19:27:49.488486 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:49.488453    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dtb2m"
Sep 15 19:27:49.488894 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:49.488538    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"
Sep 15 19:27:49.549788 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:49.549745    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"]
Sep 15 19:27:49.550006 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:49.549978    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh" podUID="553090b3-775d-46a5-9b36-efaf6a2e81df" containerName="maas-api" containerID="cri-o://d5d53d1618f8291328ecaa1f674edd44e9c47b19b01760ea1b2050ea7e7902d3" gracePeriod=30
Sep 15 19:27:49.796698 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:49.796673    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"
Sep 15 19:27:49.948217 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:49.948179    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hmv4\" (UniqueName: \"kubernetes.io/projected/553090b3-775d-46a5-9b36-efaf6a2e81df-kube-api-access-6hmv4\") pod \"553090b3-775d-46a5-9b36-efaf6a2e81df\" (UID: \"553090b3-775d-46a5-9b36-efaf6a2e81df\") "
Sep 15 19:27:49.948413 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:49.948231    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/553090b3-775d-46a5-9b36-efaf6a2e81df-maas-api-tls\") pod \"553090b3-775d-46a5-9b36-efaf6a2e81df\" (UID: \"553090b3-775d-46a5-9b36-efaf6a2e81df\") "
Sep 15 19:27:49.948413 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:49.948376    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/553090b3-775d-46a5-9b36-efaf6a2e81df-openshift-service-ca\") pod \"553090b3-775d-46a5-9b36-efaf6a2e81df\" (UID: \"553090b3-775d-46a5-9b36-efaf6a2e81df\") "
Sep 15 19:27:49.948413 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:49.948410    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/553090b3-775d-46a5-9b36-efaf6a2e81df-metrics-certs\") pod \"553090b3-775d-46a5-9b36-efaf6a2e81df\" (UID: \"553090b3-775d-46a5-9b36-efaf6a2e81df\") "
Sep 15 19:27:49.948790 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:49.948760    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/553090b3-775d-46a5-9b36-efaf6a2e81df-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "553090b3-775d-46a5-9b36-efaf6a2e81df" (UID: "553090b3-775d-46a5-9b36-efaf6a2e81df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:27:49.950924 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:49.950894    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/553090b3-775d-46a5-9b36-efaf6a2e81df-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "553090b3-775d-46a5-9b36-efaf6a2e81df" (UID: "553090b3-775d-46a5-9b36-efaf6a2e81df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:27:49.951007 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:49.950942    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/553090b3-775d-46a5-9b36-efaf6a2e81df-kube-api-access-6hmv4" (OuterVolumeSpecName: "kube-api-access-6hmv4") pod "553090b3-775d-46a5-9b36-efaf6a2e81df" (UID: "553090b3-775d-46a5-9b36-efaf6a2e81df"). InnerVolumeSpecName "kube-api-access-6hmv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:27:49.951007 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:49.950956    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/553090b3-775d-46a5-9b36-efaf6a2e81df-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "553090b3-775d-46a5-9b36-efaf6a2e81df" (UID: "553090b3-775d-46a5-9b36-efaf6a2e81df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:27:50.049440 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:50.049346    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/553090b3-775d-46a5-9b36-efaf6a2e81df-openshift-service-ca\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:27:50.049440 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:50.049379    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/553090b3-775d-46a5-9b36-efaf6a2e81df-metrics-certs\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:27:50.049440 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:50.049393    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hmv4\" (UniqueName: \"kubernetes.io/projected/553090b3-775d-46a5-9b36-efaf6a2e81df-kube-api-access-6hmv4\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:27:50.049440 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:50.049403    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/553090b3-775d-46a5-9b36-efaf6a2e81df-maas-api-tls\") on node \"ip-10-0-132-28.ec2.internal\" DevicePath \"\""
Sep 15 19:27:50.513292 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:50.513258    2555 generic.go:356] "Generic (PLEG): container finished" podID="553090b3-775d-46a5-9b36-efaf6a2e81df" containerID="d5d53d1618f8291328ecaa1f674edd44e9c47b19b01760ea1b2050ea7e7902d3" exitCode=0
Sep 15 19:27:50.513713 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:50.513306    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"
Sep 15 19:27:50.513713 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:50.513345    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh" event={"ID":"553090b3-775d-46a5-9b36-efaf6a2e81df","Type":"ContainerDied","Data":"d5d53d1618f8291328ecaa1f674edd44e9c47b19b01760ea1b2050ea7e7902d3"}
Sep 15 19:27:50.513713 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:50.513409    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh" event={"ID":"553090b3-775d-46a5-9b36-efaf6a2e81df","Type":"ContainerDied","Data":"80638ccd5902d6e60ca64025bd5abac85bcd6e5564fe6047975e7f49c07413fc"}
Sep 15 19:27:50.513713 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:50.513431    2555 scope.go:117] "RemoveContainer" containerID="d5d53d1618f8291328ecaa1f674edd44e9c47b19b01760ea1b2050ea7e7902d3"
Sep 15 19:27:50.522930 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:50.522912    2555 scope.go:117] "RemoveContainer" containerID="d5d53d1618f8291328ecaa1f674edd44e9c47b19b01760ea1b2050ea7e7902d3"
Sep 15 19:27:50.523206 ip-10-0-132-28 kubenswrapper[2555]: E0915 19:27:50.523188    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5d53d1618f8291328ecaa1f674edd44e9c47b19b01760ea1b2050ea7e7902d3\": container with ID starting with d5d53d1618f8291328ecaa1f674edd44e9c47b19b01760ea1b2050ea7e7902d3 not found: ID does not exist" containerID="d5d53d1618f8291328ecaa1f674edd44e9c47b19b01760ea1b2050ea7e7902d3"
Sep 15 19:27:50.523268 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:50.523215    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5d53d1618f8291328ecaa1f674edd44e9c47b19b01760ea1b2050ea7e7902d3"} err="failed to get container status \"d5d53d1618f8291328ecaa1f674edd44e9c47b19b01760ea1b2050ea7e7902d3\": rpc error: code = NotFound desc = could not find container \"d5d53d1618f8291328ecaa1f674edd44e9c47b19b01760ea1b2050ea7e7902d3\": container with ID starting with d5d53d1618f8291328ecaa1f674edd44e9c47b19b01760ea1b2050ea7e7902d3 not found: ID does not exist"
Sep 15 19:27:50.534213 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:50.534184    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"]
Sep 15 19:27:50.537856 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:50.537833    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-slqhh"]
Sep 15 19:27:51.040101 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:27:51.040071    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="553090b3-775d-46a5-9b36-efaf6a2e81df" path="/var/lib/kubelet/pods/553090b3-775d-46a5-9b36-efaf6a2e81df/volumes"
Sep 15 19:28:11.454939 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:11.454902    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7s247/must-gather-294xq"]
Sep 15 19:28:11.455335 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:11.455293    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="553090b3-775d-46a5-9b36-efaf6a2e81df" containerName="maas-api"
Sep 15 19:28:11.455335 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:11.455305    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="553090b3-775d-46a5-9b36-efaf6a2e81df" containerName="maas-api"
Sep 15 19:28:11.455413 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:11.455386    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="553090b3-775d-46a5-9b36-efaf6a2e81df" containerName="maas-api"
Sep 15 19:28:11.458505 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:11.458485    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7s247/must-gather-294xq"
Sep 15 19:28:11.461658 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:11.461630    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7s247\"/\"default-dockercfg-jg7th\""
Sep 15 19:28:11.461796 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:11.461765    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7s247\"/\"kube-root-ca.crt\""
Sep 15 19:28:11.462087 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:11.462061    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7s247\"/\"openshift-service-ca.crt\""
Sep 15 19:28:11.468214 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:11.468193    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7s247/must-gather-294xq"]
Sep 15 19:28:11.547247 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:11.547212    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxjz9\" (UniqueName: \"kubernetes.io/projected/c6fbe193-38fd-40b2-bcaf-47a555dde89c-kube-api-access-lxjz9\") pod \"must-gather-294xq\" (UID: \"c6fbe193-38fd-40b2-bcaf-47a555dde89c\") " pod="openshift-must-gather-7s247/must-gather-294xq"
Sep 15 19:28:11.547416 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:11.547297    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c6fbe193-38fd-40b2-bcaf-47a555dde89c-must-gather-output\") pod \"must-gather-294xq\" (UID: \"c6fbe193-38fd-40b2-bcaf-47a555dde89c\") " pod="openshift-must-gather-7s247/must-gather-294xq"
Sep 15 19:28:11.648526 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:11.648490    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxjz9\" (UniqueName: \"kubernetes.io/projected/c6fbe193-38fd-40b2-bcaf-47a555dde89c-kube-api-access-lxjz9\") pod \"must-gather-294xq\" (UID: \"c6fbe193-38fd-40b2-bcaf-47a555dde89c\") " pod="openshift-must-gather-7s247/must-gather-294xq"
Sep 15 19:28:11.648715 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:11.648655    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c6fbe193-38fd-40b2-bcaf-47a555dde89c-must-gather-output\") pod \"must-gather-294xq\" (UID: \"c6fbe193-38fd-40b2-bcaf-47a555dde89c\") " pod="openshift-must-gather-7s247/must-gather-294xq"
Sep 15 19:28:11.648995 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:11.648975    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c6fbe193-38fd-40b2-bcaf-47a555dde89c-must-gather-output\") pod \"must-gather-294xq\" (UID: \"c6fbe193-38fd-40b2-bcaf-47a555dde89c\") " pod="openshift-must-gather-7s247/must-gather-294xq"
Sep 15 19:28:11.664537 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:11.664503    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxjz9\" (UniqueName: \"kubernetes.io/projected/c6fbe193-38fd-40b2-bcaf-47a555dde89c-kube-api-access-lxjz9\") pod \"must-gather-294xq\" (UID: \"c6fbe193-38fd-40b2-bcaf-47a555dde89c\") " pod="openshift-must-gather-7s247/must-gather-294xq"
Sep 15 19:28:11.768752 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:11.768707    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7s247/must-gather-294xq"
Sep 15 19:28:12.042683 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:12.042660    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7s247/must-gather-294xq"]
Sep 15 19:28:12.044928 ip-10-0-132-28 kubenswrapper[2555]: W0915 19:28:12.044900    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6fbe193_38fd_40b2_bcaf_47a555dde89c.slice/crio-8c967a8c6dc93d8e294c8eae22abe76efc2e2258ab7a43397296d0c04f134c7d WatchSource:0}: Error finding container 8c967a8c6dc93d8e294c8eae22abe76efc2e2258ab7a43397296d0c04f134c7d: Status 404 returned error can't find the container with id 8c967a8c6dc93d8e294c8eae22abe76efc2e2258ab7a43397296d0c04f134c7d
Sep 15 19:28:12.623673 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:12.623638    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7s247/must-gather-294xq" event={"ID":"c6fbe193-38fd-40b2-bcaf-47a555dde89c","Type":"ContainerStarted","Data":"8c967a8c6dc93d8e294c8eae22abe76efc2e2258ab7a43397296d0c04f134c7d"}
Sep 15 19:28:13.634380 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:13.634333    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7s247/must-gather-294xq" event={"ID":"c6fbe193-38fd-40b2-bcaf-47a555dde89c","Type":"ContainerStarted","Data":"d5ff5f45ebafbc929e9828d84764ea234b2a535258e2e66036b777326a7aafc7"}
Sep 15 19:28:13.634893 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:13.634395    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7s247/must-gather-294xq" event={"ID":"c6fbe193-38fd-40b2-bcaf-47a555dde89c","Type":"ContainerStarted","Data":"6ec4f1af3c032b881c292bfaaa4921d69ce97e54748df70ac727152aebb2b516"}
Sep 15 19:28:13.652701 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:13.652604    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7s247/must-gather-294xq" podStartSLOduration=1.8133181870000001 podStartE2EDuration="2.652579942s" totalImagesPullingTime="839.261755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:28:13.650905985 +0000 UTC m=+2235.138234973" watchObservedRunningTime="2026-09-15 19:28:13.652579942 +0000 UTC m=+2235.139908930"
Sep 15 19:28:14.560429 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:14.560393    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j8kt6_f16c1599-2406-48bc-88fb-36e9670dc851/global-pull-secret-syncer/0.log"
Sep 15 19:28:14.793101 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:14.793065    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tdwks_cd565903-2276-469c-a07c-63cd015ef00a/konnectivity-agent/0.log"
Sep 15 19:28:14.870731 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:14.870656    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-28.ec2.internal_f8d47b24220b0dba80a2c6cc3e11210a/haproxy/0.log"
Sep 15 19:28:19.458860 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:19.458781    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-x9lmq_2779bd3b-4cdb-476b-9d11-af449b40d819/manager/0.log"
Sep 15 19:28:19.483925 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:19.483897    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-ldbhc_f4b5f5d7-9a02-4864-96f7-3cb99ecc7c97/kuadrant-console-plugin/0.log"
Sep 15 19:28:19.522115 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:19.522084    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-qzn8z_c3df758a-73f5-47dc-8c0d-a48bd65db15b/manager/0.log"
Sep 15 19:28:21.086846 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.086815    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06cd018b-7df9-4f47-a5b2-c8f9d18aa94e/alertmanager/0.log"
Sep 15 19:28:21.108446 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.108415    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06cd018b-7df9-4f47-a5b2-c8f9d18aa94e/config-reloader/0.log"
Sep 15 19:28:21.128987 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.128955    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06cd018b-7df9-4f47-a5b2-c8f9d18aa94e/kube-rbac-proxy-web/0.log"
Sep 15 19:28:21.150137 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.150111    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06cd018b-7df9-4f47-a5b2-c8f9d18aa94e/kube-rbac-proxy/0.log"
Sep 15 19:28:21.180575 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.180543    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06cd018b-7df9-4f47-a5b2-c8f9d18aa94e/kube-rbac-proxy-metric/0.log"
Sep 15 19:28:21.201879 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.201852    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06cd018b-7df9-4f47-a5b2-c8f9d18aa94e/prom-label-proxy/0.log"
Sep 15 19:28:21.226507 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.226412    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06cd018b-7df9-4f47-a5b2-c8f9d18aa94e/init-config-reloader/0.log"
Sep 15 19:28:21.306797 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.306760    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rd57n_c6910d9d-81c1-4eaa-975c-328baddaa833/kube-state-metrics/0.log"
Sep 15 19:28:21.325246 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.325141    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rd57n_c6910d9d-81c1-4eaa-975c-328baddaa833/kube-rbac-proxy-main/0.log"
Sep 15 19:28:21.356983 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.356875    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rd57n_c6910d9d-81c1-4eaa-975c-328baddaa833/kube-rbac-proxy-self/0.log"
Sep 15 19:28:21.390556 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.390531    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-57bbd7bfc6-b6nqj_d711a256-d08f-488a-ac0a-4950a9e7101e/metrics-server/0.log"
Sep 15 19:28:21.426630 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.426579    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-bqdnz_c5c66802-2cb3-4202-9ed5-f46fe621aa2b/monitoring-plugin/0.log"
Sep 15 19:28:21.593525 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.593496    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gf7cn_2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb/node-exporter/0.log"
Sep 15 19:28:21.620348 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.620253    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gf7cn_2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb/kube-rbac-proxy/0.log"
Sep 15 19:28:21.656013 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.655982    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gf7cn_2508e5c0-9cf1-4ce6-ad7d-91f55bed0deb/init-textfile/0.log"
Sep 15 19:28:21.756966 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.756933    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-w4jn7_b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf/kube-rbac-proxy-main/0.log"
Sep 15 19:28:21.778353 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.778323    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-w4jn7_b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf/kube-rbac-proxy-self/0.log"
Sep 15 19:28:21.799314 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.799278    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-w4jn7_b8ae0cf0-f0a5-428d-82bf-0509b4d94dcf/openshift-state-metrics/0.log"
Sep 15 19:28:21.836146 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.836114    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_63ee6cce-db30-41f7-bcb4-fd7d8d145013/prometheus/0.log"
Sep 15 19:28:21.858187 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.858155    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_63ee6cce-db30-41f7-bcb4-fd7d8d145013/config-reloader/0.log"
Sep 15 19:28:21.887538 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.887456    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_63ee6cce-db30-41f7-bcb4-fd7d8d145013/thanos-sidecar/0.log"
Sep 15 19:28:21.925058 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.925027    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_63ee6cce-db30-41f7-bcb4-fd7d8d145013/kube-rbac-proxy-web/0.log"
Sep 15 19:28:21.970699 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.970178    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_63ee6cce-db30-41f7-bcb4-fd7d8d145013/kube-rbac-proxy/0.log"
Sep 15 19:28:21.999060 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:21.999028    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_63ee6cce-db30-41f7-bcb4-fd7d8d145013/kube-rbac-proxy-thanos/0.log"
Sep 15 19:28:22.033640 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:22.033563    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_63ee6cce-db30-41f7-bcb4-fd7d8d145013/init-config-reloader/0.log"
Sep 15 19:28:22.069833 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:22.069798    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-x8s89_9957ce75-f34c-42b9-804a-dd68f84af8d6/prometheus-operator/0.log"
Sep 15 19:28:22.088796 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:22.088765    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-x8s89_9957ce75-f34c-42b9-804a-dd68f84af8d6/kube-rbac-proxy/0.log"
Sep 15 19:28:22.157601 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:22.157523    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8545d8745b-fkp52_695b920f-2498-4cc6-8d0e-1095adfb7f2a/telemeter-client/0.log"
Sep 15 19:28:22.176200 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:22.176160    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8545d8745b-fkp52_695b920f-2498-4cc6-8d0e-1095adfb7f2a/reload/0.log"
Sep 15 19:28:22.196147 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:22.196117    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8545d8745b-fkp52_695b920f-2498-4cc6-8d0e-1095adfb7f2a/kube-rbac-proxy/0.log"
Sep 15 19:28:22.225137 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:22.225109    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d94699b48-nfzfw_c9769660-5ff3-42a3-8df2-9202ffce4803/thanos-query/0.log"
Sep 15 19:28:22.248752 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:22.248720    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d94699b48-nfzfw_c9769660-5ff3-42a3-8df2-9202ffce4803/kube-rbac-proxy-web/0.log"
Sep 15 19:28:22.269024 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:22.268990    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d94699b48-nfzfw_c9769660-5ff3-42a3-8df2-9202ffce4803/kube-rbac-proxy/0.log"
Sep 15 19:28:22.294050 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:22.294015    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d94699b48-nfzfw_c9769660-5ff3-42a3-8df2-9202ffce4803/prom-label-proxy/0.log"
Sep 15 19:28:22.316777 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:22.316738    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d94699b48-nfzfw_c9769660-5ff3-42a3-8df2-9202ffce4803/kube-rbac-proxy-rules/0.log"
Sep 15 19:28:22.341493 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:22.341464    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d94699b48-nfzfw_c9769660-5ff3-42a3-8df2-9202ffce4803/kube-rbac-proxy-metrics/0.log"
Sep 15 19:28:24.977524 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:24.977482    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"]
Sep 15 19:28:24.982957 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:24.982930    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"
Sep 15 19:28:24.996443 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:24.994822    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"]
Sep 15 19:28:25.092917 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.092884    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvqsb\" (UniqueName: \"kubernetes.io/projected/8b202e94-2973-4c2b-9458-8dbee2c3e051-kube-api-access-gvqsb\") pod \"perf-node-gather-daemonset-xhxzz\" (UID: \"8b202e94-2973-4c2b-9458-8dbee2c3e051\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"
Sep 15 19:28:25.093114 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.092944    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b202e94-2973-4c2b-9458-8dbee2c3e051-proc\") pod \"perf-node-gather-daemonset-xhxzz\" (UID: \"8b202e94-2973-4c2b-9458-8dbee2c3e051\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"
Sep 15 19:28:25.093114 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.092987    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b202e94-2973-4c2b-9458-8dbee2c3e051-sys\") pod \"perf-node-gather-daemonset-xhxzz\" (UID: \"8b202e94-2973-4c2b-9458-8dbee2c3e051\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"
Sep 15 19:28:25.093114 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.093026    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b202e94-2973-4c2b-9458-8dbee2c3e051-lib-modules\") pod \"perf-node-gather-daemonset-xhxzz\" (UID: \"8b202e94-2973-4c2b-9458-8dbee2c3e051\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"
Sep 15 19:28:25.093306 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.093149    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b202e94-2973-4c2b-9458-8dbee2c3e051-podres\") pod \"perf-node-gather-daemonset-xhxzz\" (UID: \"8b202e94-2973-4c2b-9458-8dbee2c3e051\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"
Sep 15 19:28:25.194804 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.194769    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvqsb\" (UniqueName: \"kubernetes.io/projected/8b202e94-2973-4c2b-9458-8dbee2c3e051-kube-api-access-gvqsb\") pod \"perf-node-gather-daemonset-xhxzz\" (UID: \"8b202e94-2973-4c2b-9458-8dbee2c3e051\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"
Sep 15 19:28:25.195005 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.194846    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b202e94-2973-4c2b-9458-8dbee2c3e051-proc\") pod \"perf-node-gather-daemonset-xhxzz\" (UID: \"8b202e94-2973-4c2b-9458-8dbee2c3e051\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"
Sep 15 19:28:25.195005 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.194910    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b202e94-2973-4c2b-9458-8dbee2c3e051-sys\") pod \"perf-node-gather-daemonset-xhxzz\" (UID: \"8b202e94-2973-4c2b-9458-8dbee2c3e051\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"
Sep 15 19:28:25.195005 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.194964    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b202e94-2973-4c2b-9458-8dbee2c3e051-lib-modules\") pod \"perf-node-gather-daemonset-xhxzz\" (UID: \"8b202e94-2973-4c2b-9458-8dbee2c3e051\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"
Sep 15 19:28:25.195005 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.195004    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b202e94-2973-4c2b-9458-8dbee2c3e051-podres\") pod \"perf-node-gather-daemonset-xhxzz\" (UID: \"8b202e94-2973-4c2b-9458-8dbee2c3e051\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"
Sep 15 19:28:25.195219 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.195162    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b202e94-2973-4c2b-9458-8dbee2c3e051-podres\") pod \"perf-node-gather-daemonset-xhxzz\" (UID: \"8b202e94-2973-4c2b-9458-8dbee2c3e051\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"
Sep 15 19:28:25.195219 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.195194    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b202e94-2973-4c2b-9458-8dbee2c3e051-sys\") pod \"perf-node-gather-daemonset-xhxzz\" (UID: \"8b202e94-2973-4c2b-9458-8dbee2c3e051\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"
Sep 15 19:28:25.195325 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.195233    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b202e94-2973-4c2b-9458-8dbee2c3e051-lib-modules\") pod \"perf-node-gather-daemonset-xhxzz\" (UID: \"8b202e94-2973-4c2b-9458-8dbee2c3e051\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"
Sep 15 19:28:25.195325 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.195260    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b202e94-2973-4c2b-9458-8dbee2c3e051-proc\") pod \"perf-node-gather-daemonset-xhxzz\" (UID: \"8b202e94-2973-4c2b-9458-8dbee2c3e051\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"
Sep 15 19:28:25.203441 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.203374    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvqsb\" (UniqueName: \"kubernetes.io/projected/8b202e94-2973-4c2b-9458-8dbee2c3e051-kube-api-access-gvqsb\") pod \"perf-node-gather-daemonset-xhxzz\" (UID: \"8b202e94-2973-4c2b-9458-8dbee2c3e051\") " pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"
Sep 15 19:28:25.297386 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.297345    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"
Sep 15 19:28:25.594243 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.594206    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"]
Sep 15 19:28:25.726544 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.726497    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz" event={"ID":"8b202e94-2973-4c2b-9458-8dbee2c3e051","Type":"ContainerStarted","Data":"aa2d74c874b324891259f96204ed0dabd39828383766665711e6d1f9eda4c0bc"}
Sep 15 19:28:25.726930 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.726629    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz" event={"ID":"8b202e94-2973-4c2b-9458-8dbee2c3e051","Type":"ContainerStarted","Data":"0a174cf791717692f5a3962121fc9c40fe6dfd2bc50ebf2ed8f9971a22970786"}
Sep 15 19:28:25.726930 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.726886    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"
Sep 15 19:28:25.758690 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:25.758623    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz" podStartSLOduration=1.758580458 podStartE2EDuration="1.758580458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 19:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:28:25.75584698 +0000 UTC m=+2247.243175967" watchObservedRunningTime="2026-09-15 19:28:25.758580458 +0000 UTC m=+2247.245909445"
Sep 15 19:28:27.043492 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:27.043461    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rldqf_48b28e4d-19dd-43ea-ae96-cd04ac848020/dns/0.log"
Sep 15 19:28:27.064360 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:27.064333    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rldqf_48b28e4d-19dd-43ea-ae96-cd04ac848020/kube-rbac-proxy/0.log"
Sep 15 19:28:27.109320 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:27.109286    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d8r6c_c9555dd5-d766-4b43-bd0b-e2ed73acc9cd/dns-node-resolver/0.log"
Sep 15 19:28:27.710602 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:27.710573    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g2fwg_4824dade-a75e-4ddc-b086-989eaedafe12/node-ca/0.log"
Sep 15 19:28:29.087506 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:29.087469    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b76b67bf4-xtqs9_240e4169-e142-49f2-9f8d-110075a03bc6/kube-auth-proxy/0.log"
Sep 15 19:28:30.107563 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:30.107530    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zknv8_9c998a93-b454-41f5-a589-87f5364c062c/serve-healthcheck-canary/0.log"
Sep 15 19:28:30.713846 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:30.713818    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wh7k8_afa9ab3a-3301-4822-b409-2cbbdb957b28/kube-rbac-proxy/0.log"
Sep 15 19:28:30.734530 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:30.734491    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wh7k8_afa9ab3a-3301-4822-b409-2cbbdb957b28/exporter/0.log"
Sep 15 19:28:30.761283 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:30.761239    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wh7k8_afa9ab3a-3301-4822-b409-2cbbdb957b28/extractor/0.log"
Sep 15 19:28:31.743022 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:31.742993    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7s247/perf-node-gather-daemonset-xhxzz"
Sep 15 19:28:32.881630 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:32.881583    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7f6f96c899-chqcs_a415ba45-bfde-4acc-9544-c1dab8ab93a0/manager/0.log"
Sep 15 19:28:33.483803 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:33.483755    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7467f6b4cf-crsfx_a5b931e3-1eb1-4272-b43c-29d12313a10a/manager/0.log"
Sep 15 19:28:33.508135 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:33.508095    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7467f6b4cf-crsfx_a5b931e3-1eb1-4272-b43c-29d12313a10a/copy-manifests/0.log"
Sep 15 19:28:35.223072 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:35.223030    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5cb45c66cf-w9dtj_4fb0e9a9-273b-455e-9f54-227f9e19c380/manager/0.log"
Sep 15 19:28:35.251167 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:35.251118    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-b5zgz_af171b3e-d379-4f04-9589-d43a1d33a9c9/openshift-lws-operator/0.log"
Sep 15 19:28:35.811699 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:35.811669    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b76b67bf4-xtqs9_240e4169-e142-49f2-9f8d-110075a03bc6/kube-auth-proxy/0.log"
Sep 15 19:28:37.745158 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:37.745121    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b76b67bf4-xtqs9_240e4169-e142-49f2-9f8d-110075a03bc6/kube-auth-proxy/0.log"
Sep 15 19:28:41.625164 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:41.625132    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2sq4v_598c3d78-c05f-42dc-ac4f-0a6640c6fc38/kube-multus-additional-cni-plugins/0.log"
Sep 15 19:28:41.644283 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:41.644248    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2sq4v_598c3d78-c05f-42dc-ac4f-0a6640c6fc38/egress-router-binary-copy/0.log"
Sep 15 19:28:41.665006 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:41.664977    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2sq4v_598c3d78-c05f-42dc-ac4f-0a6640c6fc38/cni-plugins/0.log"
Sep 15 19:28:41.684396 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:41.684370    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2sq4v_598c3d78-c05f-42dc-ac4f-0a6640c6fc38/bond-cni-plugin/0.log"
Sep 15 19:28:41.703517 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:41.703486    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2sq4v_598c3d78-c05f-42dc-ac4f-0a6640c6fc38/routeoverride-cni/0.log"
Sep 15 19:28:41.723829 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:41.723804    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2sq4v_598c3d78-c05f-42dc-ac4f-0a6640c6fc38/whereabouts-cni-bincopy/0.log"
Sep 15 19:28:41.743655 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:41.743630    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2sq4v_598c3d78-c05f-42dc-ac4f-0a6640c6fc38/whereabouts-cni/0.log"
Sep 15 19:28:42.184119 ip-10-0-132-28 kubenswrapper[2555]: I0915 19:28:42.183938    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xfzjp_5cf60c84-6ebf-48cc-b094-f865d877bc03/kube-multus/0.log"