Sep 10 15:40:02.378146 ip-10-0-134-36 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 15:40:02.378158 ip-10-0-134-36 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 15:40:02.378168 ip-10-0-134-36 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 15:40:02.378388 ip-10-0-134-36 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 15:40:13.784624 ip-10-0-134-36 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 15:40:13.784638 ip-10-0-134-36 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e3e5a7ebc34c44a8b35f742a4b7a79ec --
Sep 10 15:42:06.151719 ip-10-0-134-36 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 15:42:06.545187 ip-10-0-134-36 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 15:42:06.545187 ip-10-0-134-36 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 15:42:06.545187 ip-10-0-134-36 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 15:42:06.545187 ip-10-0-134-36 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 15:42:06.545187 ip-10-0-134-36 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 15:42:06.548057 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.547958    2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 15:42:06.552120 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552100    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:42:06.552120 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552119    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:42:06.552120 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552124    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:42:06.552234 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552127    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:42:06.552234 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552133    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:42:06.552234 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552138    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:42:06.552234 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552141    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:42:06.552234 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552144    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:42:06.552234 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552147    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:42:06.552234 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552150    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:42:06.552234 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552153    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:42:06.552234 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552156    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:42:06.552234 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552158    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:42:06.552234 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552161    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:42:06.552234 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552170    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:42:06.552234 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552173    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:42:06.552234 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552176    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:42:06.552234 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552179    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:42:06.552234 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552182    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:42:06.552234 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552184    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:42:06.552234 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552187    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:42:06.552234 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552190    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552192    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552195    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552197    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552200    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552203    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552206    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552208    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552211    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552214    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552216    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552219    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552223    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552225    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552228    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552232    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552235    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552239    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552242    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552245    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:42:06.552712 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552250    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:42:06.553191 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552254    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:42:06.553191 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552257    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:42:06.553191 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552261    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:42:06.553191 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552264    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:42:06.553191 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552267    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:42:06.553191 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552269    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:42:06.553191 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552272    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:42:06.553191 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552275    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:42:06.553191 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552277    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:42:06.553191 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552281    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:42:06.553191 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552283    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:42:06.553191 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552286    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:42:06.553191 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552288    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:42:06.553191 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552291    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:42:06.553191 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552294    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:42:06.553191 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552297    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:42:06.553191 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552299    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:42:06.553191 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552302    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:42:06.553191 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552304    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552307    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552309    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552312    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552314    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552316    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552319    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552321    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552325    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552328    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552330    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552333    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552336    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552339    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552342    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552344    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552346    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552352    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552354    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552356    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:42:06.553688 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552359    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:42:06.554257 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552361    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:42:06.554257 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552364    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:42:06.554257 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552367    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:42:06.554257 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552369    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:42:06.554257 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552372    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:42:06.554257 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552374    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:42:06.554257 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552377    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:42:06.554257 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552381    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:42:06.554257 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552385    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:42:06.554257 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552388    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:42:06.554257 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552391    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:42:06.554257 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552408    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:42:06.554257 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552411    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:42:06.554257 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552414    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:42:06.554257 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552417    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:42:06.554257 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552421    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:42:06.554257 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552424    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:42:06.554257 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.552427    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:42:06.554257 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553148    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553156    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553159    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553161    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553166    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553169    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553172    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553174    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553177    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553180    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553183    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553185    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553188    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553190    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553193    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553196    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553199    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553201    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553204    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553207    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:42:06.554755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553210    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:42:06.555246 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553212    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:42:06.555246 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553215    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:42:06.555246 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553218    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:42:06.555246 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553221    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:42:06.555246 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553223    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:42:06.555246 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553226    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:42:06.555246 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553228    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:42:06.555246 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553231    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:42:06.555246 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553234    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:42:06.555246 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553237    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:42:06.555246 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553239    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:42:06.555246 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553242    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:42:06.555246 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553246    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:42:06.555246 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553248    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:42:06.555246 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553251    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:42:06.555246 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553253    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:42:06.555246 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553256    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:42:06.555246 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553262    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:42:06.555246 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553265    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:42:06.555728 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553267    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:42:06.555728 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553273    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:42:06.555728 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553276    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:42:06.555728 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553279    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:42:06.555728 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553282    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:42:06.555728 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553286    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:42:06.555728 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553288    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:42:06.555728 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553291    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:42:06.555728 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553294    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:42:06.555728 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553296    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:42:06.555728 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553299    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:42:06.555728 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553302    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:42:06.555728 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553304    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:42:06.555728 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553306    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:42:06.555728 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553309    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:42:06.555728 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553312    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:42:06.555728 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553314    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:42:06.555728 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553317    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:42:06.555728 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553319    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553321    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553324    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553327    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553330    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553333    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553339    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553345    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553348    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553350    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553353    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553356    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553359    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553362    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553365    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553367    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553370    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553373    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553375    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553378    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:42:06.556195 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553380    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:42:06.556722 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553383    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:42:06.556722 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553385    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:42:06.556722 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553388    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:42:06.556722 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553390    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:42:06.556722 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553393    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:42:06.556722 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553417    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:42:06.556722 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553420    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:42:06.556722 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553422    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:42:06.556722 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553425    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:42:06.556722 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553428    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:42:06.556722 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553431    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:42:06.556722 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553434    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:42:06.556722 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553436    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:42:06.556722 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553439    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:42:06.556722 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553443    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:42:06.556722 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553446    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:42:06.556722 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553449    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:42:06.556722 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553453    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:42:06.556722 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553456    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:42:06.557218 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553459    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:42:06.557218 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553897    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:42:06.557218 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553903    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:42:06.557218 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553906    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:42:06.557218 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553910    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:42:06.557218 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553913    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:42:06.557218 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553918    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:42:06.557218 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553922    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:42:06.557218 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553925    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:42:06.557218 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553928    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:42:06.557218 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553931    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:42:06.557218 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553935    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:42:06.557218 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553939    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:42:06.557218 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553941    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:42:06.557218 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553944    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:42:06.557218 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553947    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:42:06.557218 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553950    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:42:06.557218 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553953    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:42:06.557218 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553956    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553958    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553961    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553963    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553966    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553968    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553971    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553974    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553976    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553979    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553982    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553984    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553988    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553993    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553995    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.553998    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554000    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554003    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554006    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554008    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:42:06.557707 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554012    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554015    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554018    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554020    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554023    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554026    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554029    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554031    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554034    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554037    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554039    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554041    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554044    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554047    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554050    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554052    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554055    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554057    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554060    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554063    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:42:06.558236 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554065    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554068    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554070    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554073    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554077    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554080    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554082    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554085    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554088    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554090    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554093    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554095    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554098    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554101    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554103    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554106    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554108    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554111    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554113    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554117    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:42:06.558740 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554120    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:42:06.559227 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554123    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:42:06.559227 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554125    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:42:06.559227 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554128    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:42:06.559227 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554131    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:42:06.559227 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554133    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:42:06.559227 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554136    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:42:06.559227 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554139    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:42:06.559227 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554142    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:42:06.559227 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554146    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:42:06.559227 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554148    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:42:06.559227 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554152    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:42:06.559227 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554154    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:42:06.559227 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554157    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:42:06.559227 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554160    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:42:06.559227 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554163    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:42:06.559227 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554167    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:42:06.559227 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554170    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:42:06.559227 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554173    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:42:06.559227 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554175    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554178    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554180    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554254    2572 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554284    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554291    2572 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554296    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554301    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554305    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554310    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554315    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554318    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554321    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554325    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554328    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554331    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554334    2572 flags.go:64] FLAG: --cgroup-root=""
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554337    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554340    2572 flags.go:64] FLAG: --client-ca-file=""
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554343    2572 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554347    2572 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554351    2572 flags.go:64] FLAG: --cluster-domain=""
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554353    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554357    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 15:42:06.559797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554360    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554363    2572 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554367    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554370    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554373    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554376    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554380    2572 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554383    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554386    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554389    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554392    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554410    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554413    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554416    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554419    2572 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554422    2572 flags.go:64] FLAG: --enable-server="true"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554425    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554430    2572 flags.go:64] FLAG: --event-burst="100"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554433    2572 flags.go:64] FLAG: --event-qps="50"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554436    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554439    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554442    2572 flags.go:64] FLAG: --eviction-hard=""
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554445    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554448    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554451    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 15:42:06.560384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554455    2572 flags.go:64] FLAG: --eviction-soft=""
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554458    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554461    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554463    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554466    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554469    2572 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554472    2572 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554475    2572 flags.go:64] FLAG: --feature-gates=""
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554479    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554482    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554485    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554488    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554491    2572 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554496    2572 flags.go:64] FLAG: --help="false"
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554499    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-134-36.ec2.internal"
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554503    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554506    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554509    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554513    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554516    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554519    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554522    2572 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554525    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 15:42:06.561002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554528    2572 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554532    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554535    2572 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554538    2572 flags.go:64] FLAG: --kube-reserved=""
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554541    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554544    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554547    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554550    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554552    2572 flags.go:64] FLAG: --lock-file=""
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554555    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554558    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554561    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554567    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554570    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554573    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554575    2572 flags.go:64] FLAG: --logging-format="text"
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554578    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554582    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554585    2572 flags.go:64] FLAG: --manifest-url=""
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554588    2572 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554592    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554595    2572 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554601    2572 flags.go:64] FLAG: --max-pods="110"
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554604    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554607    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 15:42:06.561627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554610    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554613    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554616    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554619    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554622    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554630    2572 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554633    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554636    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554639    2572 flags.go:64] FLAG: --pod-cidr=""
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554642    2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554648    2572 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554651    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554655    2572 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554658    2572 flags.go:64] FLAG: --port="10250"
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554661    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554663    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00d1ec3d5a8d8ad0d"
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554666    2572 flags.go:64] FLAG: --qos-reserved=""
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554669    2572 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554672    2572 flags.go:64] FLAG: --register-node="true"
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554675    2572 flags.go:64] FLAG: --register-with-taints=""
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554678    2572 flags.go:64] FLAG: --registry-burst="10"
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554681    2572 flags.go:64] FLAG: --registry-qps="5"
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554684    2572 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554687    2572 flags.go:64] FLAG: --reserved-memory=""
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554691    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 15:42:06.562241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554694    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554697    2572 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554700    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554703    2572 flags.go:64] FLAG: --runonce="false"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554707    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554710    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554713    2572 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554716    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554719    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554722    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554725    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554728    2572 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554731    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554734    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554737    2572 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554740    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554743    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554747    2572 flags.go:64] FLAG: --system-cgroups=""
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554750    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554755    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554758    2572 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554761    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554766    2572 flags.go:64] FLAG: --tls-min-version=""
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554768    2572 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554771    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 15:42:06.562884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554775    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554778    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554780    2572 flags.go:64] FLAG: --v="2"
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554785    2572 flags.go:64] FLAG: --version="false"
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554789    2572 flags.go:64] FLAG: --vmodule=""
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554793    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.554796    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554910    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554915    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554919    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554921    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554925    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554928    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554930    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554933    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554936    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554938    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554941    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554943    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554945    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554948    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554951    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:42:06.563558 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554953    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554956    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554958    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554963    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554965    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554968    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554971    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554973    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554975    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554978    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554981    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554983    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554985    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554988    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554990    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554993    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554995    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.554998    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555002    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555004    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:42:06.564101 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555007    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:42:06.564619 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555013    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:42:06.564619 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555016    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:42:06.564619 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555020    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:42:06.564619 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555023    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:42:06.564619 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555026    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:42:06.564619 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555029    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:42:06.564619 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555032    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:42:06.564619 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555034    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:42:06.564619 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555037    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:42:06.564619 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555040    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:42:06.564619 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555043    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:42:06.564619 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555046    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:42:06.564619 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555049    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:42:06.564619 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555051    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:42:06.564619 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555054    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:42:06.564619 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555057    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:42:06.564619 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555060    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:42:06.564619 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555063    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:42:06.564619 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555066    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:42:06.565083 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555069    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:42:06.565083 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555072    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:42:06.565083 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555074    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:42:06.565083 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555077    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:42:06.565083 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555080    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:42:06.565083 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555082    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:42:06.565083 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555084    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:42:06.565083 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555087    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:42:06.565083 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555090    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:42:06.565083 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555093    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:42:06.565083 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555096    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:42:06.565083 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555099    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:42:06.565083 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555102    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:42:06.565083 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555106    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:42:06.565083 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555109    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:42:06.565083 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555112    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:42:06.565083 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555114    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:42:06.565083 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555117    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:42:06.565083 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555120    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555122    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555125    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555127    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555130    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555132    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555135    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555137    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555141    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555144    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555147    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555150    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555153    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555155    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555158    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555160    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555163    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555165    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555168    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555170    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:42:06.565593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555173    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:42:06.566096 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555175    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:42:06.566096 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555178    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:42:06.566096 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555180    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:42:06.566096 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555184    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:42:06.566096 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.555187    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:42:06.566096 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.556114    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 15:42:06.566096 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.564351    2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 15:42:06.566096 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.564376    2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 15:42:06.566096 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.564410    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 15:42:06.566096 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.564417    2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 15:42:06.566096 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564484    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:42:06.566096 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564490    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:42:06.566096 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564493    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:42:06.566096 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564496    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564499    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564501    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564504    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564507    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564510    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564513    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564515    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564518    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564521    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564524    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564527    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564529    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564532    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564534    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564537    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564540    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564542    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564546    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564551    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:42:06.566466 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564554    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:42:06.566988 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564556    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:42:06.566988 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564559    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:42:06.566988 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564561    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:42:06.566988 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564564    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:42:06.566988 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564567    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:42:06.566988 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564577    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:42:06.566988 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564580    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:42:06.566988 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564582    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:42:06.566988 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564585    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:42:06.566988 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564589    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:42:06.566988 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564593    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:42:06.566988 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564596    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:42:06.566988 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564599    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:42:06.566988 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564602    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:42:06.566988 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564605    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:42:06.566988 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564608    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:42:06.566988 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564611    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:42:06.566988 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564615    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:42:06.566988 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564618    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:42:06.567553 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564621    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:42:06.567553 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564624    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:42:06.567553 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564627    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:42:06.567553 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564630    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:42:06.567553 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564633    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:42:06.567553 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564636    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:42:06.567553 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564638    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:42:06.567553 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564641    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:42:06.567553 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564644    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:42:06.567553 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564646    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:42:06.567553 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564649    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:42:06.567553 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564652    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:42:06.567553 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564655    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:42:06.567553 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564658    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:42:06.567553 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564660    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:42:06.567553 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564663    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:42:06.567553 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564665    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:42:06.567553 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564668    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:42:06.567553 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564671    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564679    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564682    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564685    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564688    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564691    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564693    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564696    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564698    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564701    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564704    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564706    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564709    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564717    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564720    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564722    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564725    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564727    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564730    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564733    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:42:06.568015 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564735    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:42:06.568527 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564738    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:42:06.568527 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564741    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:42:06.568527 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564744    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:42:06.568527 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564747    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:42:06.568527 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564749    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:42:06.568527 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564752    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:42:06.568527 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564755    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:42:06.568527 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564757    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:42:06.568527 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564760    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:42:06.568527 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564763    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:42:06.568527 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564765    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:42:06.568527 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564768    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:42:06.568527 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564771    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:42:06.568527 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564779    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:42:06.568527 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564782    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:42:06.568527 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564784    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:42:06.568527 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564787    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:42:06.568946 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.564792    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 15:42:06.568946 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564960    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:42:06.568946 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564968    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:42:06.568946 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564971    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:42:06.568946 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564976    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:42:06.568946 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564980    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:42:06.568946 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564983    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:42:06.568946 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564986    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:42:06.568946 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564989    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:42:06.568946 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564992    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:42:06.568946 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564994    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:42:06.568946 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.564997    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:42:06.568946 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565000    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:42:06.568946 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565003    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:42:06.568946 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565005    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:42:06.569321 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565008    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:42:06.569321 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565010    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:42:06.569321 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565013    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:42:06.569321 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565016    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:42:06.569321 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565019    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:42:06.569321 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565021    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:42:06.569321 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565024    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:42:06.569321 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565026    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:42:06.569321 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565029    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:42:06.569321 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565031    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:42:06.569321 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565034    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:42:06.569321 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565036    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:42:06.569321 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565039    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:42:06.569321 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565041    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:42:06.569321 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565049    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:42:06.569321 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565052    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:42:06.569321 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565055    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:42:06.569321 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565057    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:42:06.569321 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565060    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565062    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565065    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565067    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565070    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565072    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565075    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565077    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565080    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565082    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565085    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565087    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565090    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565093    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565095    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565098    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565101    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565104    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565106    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565109    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:42:06.569808 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565111    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565114    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565116    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565119    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565121    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565124    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565127    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565129    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565132    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565140    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565143    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565146    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565148    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565151    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565153    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565156    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565158    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565160    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565163    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565165    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:42:06.570288 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565168    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:42:06.570790 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565170    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:42:06.570790 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565173    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:42:06.570790 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565175    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:42:06.570790 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565178    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:42:06.570790 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565180    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:42:06.570790 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565183    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:42:06.570790 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565185    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:42:06.570790 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565188    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:42:06.570790 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565192    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:42:06.570790 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565195    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:42:06.570790 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565198    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:42:06.570790 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565201    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:42:06.570790 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565204    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:42:06.570790 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565206    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:42:06.570790 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565210    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:42:06.570790 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565213    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:42:06.570790 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565216    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:42:06.570790 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565219    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:42:06.570790 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565222    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:42:06.571251 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565224    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:42:06.571251 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565227    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:42:06.571251 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565235    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:42:06.571251 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565238    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:42:06.571251 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565240    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:42:06.571251 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565243    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:42:06.571251 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:06.565245    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:42:06.571251 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.565251    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 15:42:06.571251 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.565426    2572 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 15:42:06.571251 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.568021    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 15:42:06.571251 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.569095    2572 server.go:1019] "Starting client certificate rotation"
Sep 10 15:42:06.571251 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.569224    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 15:42:06.571251 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.570141    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 15:42:06.572105 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.572089    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 15:42:06.575605 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.575579    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 15:42:06.587941 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.587917    2572 log.go:25] "Validated CRI v1 runtime API"
Sep 10 15:42:06.593071 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.593054    2572 log.go:25] "Validated CRI v1 image API"
Sep 10 15:42:06.594012 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.593988    2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 15:42:06.598321 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.598299    2572 fs.go:135] Filesystem UUIDs: map[6840a622-33e6-4ee2-ae8e-9d3d1fca8639:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f50d6ac7-db46-4d22-b992-c5c02fae9db8:/dev/nvme0n1p4]
Sep 10 15:42:06.598418 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.598319    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 15:42:06.602690 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.602567    2572 manager.go:216] Machine: {Timestamp:2026-09-10 15:42:06.602057449 +0000 UTC m=+0.355371390 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098883 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f39354f926f1052782a6ac9144216 SystemUUID:ec2f3935-4f92-6f10-5278-2a6ac9144216 BootID:e3e5a7eb-c34c-44a8-b35f-742a4b7a79ec Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:06:09:3e:1d:bb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:06:09:3e:1d:bb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:81:12:a2:17:1c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 15:42:06.602690 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.602678    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 15:42:06.602825 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.602765    2572 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 15:42:06.604644 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.604627    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 15:42:06.606506 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.606474    2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 15:42:06.606656 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.606511    2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-36.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 15:42:06.607414 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.607404    2572 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 15:42:06.607452 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.607416    2572 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 15:42:06.607452 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.607424    2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 15:42:06.609200 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.609188    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 15:42:06.611039 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.611026    2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 15:42:06.616189 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.616173    2572 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 15:42:06.616320 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.616311    2572 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 15:42:06.616357 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.616345    2572 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 15:42:06.616357 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.616355    2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 15:42:06.616426 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.616371    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 15:42:06.616426 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.616382    2572 kubelet.go:398] "Adding apiserver pod source"
Sep 10 15:42:06.616426 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.616410    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 15:42:06.616872 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.616857    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 15:42:06.616939 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.616882    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 15:42:06.619538 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.619519    2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 15:42:06.624443 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.624392    2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 15:42:06.624520 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.624458    2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 15:42:06.624661 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.624650    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 15:42:06.624707 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.624667    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 15:42:06.624707 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.624673    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 15:42:06.624707 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.624679    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 15:42:06.624707 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.624687    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 15:42:06.624707 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.624692    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 15:42:06.625294 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.625285    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 15:42:06.625329 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.625295    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 15:42:06.625329 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.625305    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 15:42:06.625329 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.625313    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 15:42:06.625329 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.625323    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 15:42:06.625467 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.625332    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 15:42:06.625467 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.625356    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 15:42:06.625467 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.625361    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 15:42:06.627165 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:06.627141    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-36.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 15:42:06.627340 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:06.627314    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 15:42:06.629440 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.629416    2572 server.go:1268] "Started kubelet"
Sep 10 15:42:06.629539 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.629472    2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 15:42:06.629539 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.629499    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 15:42:06.629651 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.629564    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 15:42:06.630382 ip-10-0-134-36 systemd[1]: Started Kubernetes Kubelet.
Sep 10 15:42:06.631079 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.631059    2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 15:42:06.631754 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.631737    2572 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 15:42:06.634380 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:06.634356    2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 15:42:06.634820 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.634798    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ms5m2"
Sep 10 15:42:06.636377 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.636362    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-36.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 15:42:06.638365 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:06.636302    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-36.ec2.internal.18d3fffa1967da72  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-36.ec2.internal,UID:ip-10-0-134-36.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-36.ec2.internal,},FirstTimestamp:2026-09-10 15:42:06.62936229 +0000 UTC m=+0.382676233,LastTimestamp:2026-09-10 15:42:06.62936229 +0000 UTC m=+0.382676233,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-36.ec2.internal,}"
Sep 10 15:42:06.638480 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.638423    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 15:42:06.639116 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.639098    2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 15:42:06.639207 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:06.639196    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-36.ec2.internal\" not found"
Sep 10 15:42:06.639958 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.639936    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 15:42:06.639958 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.639937    2572 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 15:42:06.640087 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.639977    2572 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 15:42:06.640087 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.640080    2572 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 15:42:06.640087 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.640088    2572 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 15:42:06.640214 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.640101    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 15:42:06.640214 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.640111    2572 factory.go:55] Registering systemd factory
Sep 10 15:42:06.640214 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.640117    2572 factory.go:223] Registration of the systemd container factory successfully
Sep 10 15:42:06.640332 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.640312    2572 factory.go:153] Registering CRI-O factory
Sep 10 15:42:06.640332 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.640323    2572 factory.go:223] Registration of the crio container factory successfully
Sep 10 15:42:06.640426 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.640340    2572 factory.go:103] Registering Raw factory
Sep 10 15:42:06.640426 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.640352    2572 manager.go:1196] Started watching for new ooms in manager
Sep 10 15:42:06.640638 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.640623    2572 manager.go:349] Starting recovery of all containers
Sep 10 15:42:06.642217 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.642198    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ms5m2"
Sep 10 15:42:06.643995 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:06.643955    2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-36.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 15:42:06.644118 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:06.644096    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 15:42:06.653362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.653339    2572 manager.go:354] Recovery completed
Sep 10 15:42:06.658046 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.657916    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:06.660147 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.660129    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:06.660229 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.660161    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:06.660229 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.660175    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:06.660749 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.660734    2572 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 15:42:06.660749 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.660748    2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 15:42:06.660862 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.660765    2572 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 15:42:06.660862 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.660820    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:06.661599 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.661581    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:06.661664 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.661614    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:06.661664 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.661629    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:06.661784 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.661771    2572 policy_none.go:49] "None policy: Start"
Sep 10 15:42:06.661814 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.661790    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 15:42:06.661814 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.661805    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 15:42:06.662840 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.662828    2572 policy_none.go:47] "Start"
Sep 10 15:42:06.689039 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.689019    2572 manager.go:337] "Starting Device Plugin manager"
Sep 10 15:42:06.689174 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:06.689060    2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 15:42:06.689174 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.689073    2572 server.go:87] "Starting device plugin registration server"
Sep 10 15:42:06.689522 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.689489    2572 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 15:42:06.689522 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.689505    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 15:42:06.689657 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.689639    2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 15:42:06.689751 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.689734    2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 15:42:06.689751 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.689747    2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 15:42:06.690632 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:06.690610    2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 15:42:06.690718 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:06.690671    2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-36.ec2.internal\" not found"
Sep 10 15:42:06.703621 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.703604    2572 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2604 into the system container \"/system.slice\": <nil>"
Sep 10 15:42:06.790593 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.790564    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:06.791809 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.791789    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:06.791919 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.791822    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:06.791919 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.791837    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:06.791919 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.791871    2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-36.ec2.internal"
Sep 10 15:42:06.800169 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.800074    2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-36.ec2.internal"
Sep 10 15:42:06.800169 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:06.800106    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-36.ec2.internal\": node \"ip-10-0-134-36.ec2.internal\" not found"
Sep 10 15:42:06.829620 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:06.829587    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-36.ec2.internal\" not found"
Sep 10 15:42:06.903561 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.903527    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 15:42:06.904731 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.904714    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 15:42:06.904787 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.904769    2572 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 15:42:06.904833 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.904799    2572 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 15:42:06.904906 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:06.904892    2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 15:42:06.907186 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:06.907161    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:42:06.930331 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:06.930300    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-36.ec2.internal\" not found"
Sep 10 15:42:07.005816 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.005783    2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-36.ec2.internal"]
Sep 10 15:42:07.005906 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.005886    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:07.006893 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.006878    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:07.006956 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.006904    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:07.006956 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.006932    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:07.008208 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.008196    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:07.008341 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.008328    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal"
Sep 10 15:42:07.008376 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.008362    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:07.008945 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.008930    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:07.009003 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.008958    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:07.009003 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.008969    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:07.009060 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.008937    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:07.009060 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.009025    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:07.009060 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.009039    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:07.010237 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.010224    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-36.ec2.internal"
Sep 10 15:42:07.010290 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.010249    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:07.010884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.010868    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:07.010963 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.010895    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:07.010963 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.010908    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:07.031117 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:07.031084    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-36.ec2.internal\" not found"
Sep 10 15:42:07.034123 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:07.034104    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-36.ec2.internal\" not found" node="ip-10-0-134-36.ec2.internal"
Sep 10 15:42:07.038529 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:07.038511    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-36.ec2.internal\" not found" node="ip-10-0-134-36.ec2.internal"
Sep 10 15:42:07.042551 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.042533    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/872387aaa69ba3ec01d1d25ea3dd5b1d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-36.ec2.internal\" (UID: \"872387aaa69ba3ec01d1d25ea3dd5b1d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-36.ec2.internal"
Sep 10 15:42:07.042635 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.042563    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f87d7a97be63e6b3b27b586fea74839e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal\" (UID: \"f87d7a97be63e6b3b27b586fea74839e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal"
Sep 10 15:42:07.042635 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.042588    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f87d7a97be63e6b3b27b586fea74839e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal\" (UID: \"f87d7a97be63e6b3b27b586fea74839e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal"
Sep 10 15:42:07.131375 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:07.131285    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-36.ec2.internal\" not found"
Sep 10 15:42:07.143752 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.143722    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f87d7a97be63e6b3b27b586fea74839e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal\" (UID: \"f87d7a97be63e6b3b27b586fea74839e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal"
Sep 10 15:42:07.143865 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.143760    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f87d7a97be63e6b3b27b586fea74839e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal\" (UID: \"f87d7a97be63e6b3b27b586fea74839e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal"
Sep 10 15:42:07.143865 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.143784    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/872387aaa69ba3ec01d1d25ea3dd5b1d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-36.ec2.internal\" (UID: \"872387aaa69ba3ec01d1d25ea3dd5b1d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-36.ec2.internal"
Sep 10 15:42:07.143865 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.143812    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f87d7a97be63e6b3b27b586fea74839e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal\" (UID: \"f87d7a97be63e6b3b27b586fea74839e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal"
Sep 10 15:42:07.143865 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.143823    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f87d7a97be63e6b3b27b586fea74839e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal\" (UID: \"f87d7a97be63e6b3b27b586fea74839e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal"
Sep 10 15:42:07.143865 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:07.143817    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/872387aaa69ba3ec01d1d25ea3dd5b1d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-36.ec2.internal\" (UID: \"872387aaa69ba3ec01d1d25ea3dd5b1d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-36.ec2.internal"
Sep 10 15:42:07.231931 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:07.231890    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-36.ec2.internal\" not found"
Sep 10 15:42:08.413346 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:08.413279    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-36.ec2.internal\" not found"
Sep 10 15:42:08.417434 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.417417    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal"
Sep 10 15:42:08.422405 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.422386    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-36.ec2.internal"
Sep 10 15:42:08.514238 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:08.514146    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-36.ec2.internal\" not found"
Sep 10 15:42:08.531829 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.531799    2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:42:08.621154 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.621126    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal"
Sep 10 15:42:08.632345 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.632315    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 15:42:08.633650 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.633635    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-36.ec2.internal"
Sep 10 15:42:08.640661 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.640648    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 15:42:08.649809 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.649790    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 15:42:08.649946 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.649930    2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 15:42:08.649980 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.649946    2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 15:42:08.698124 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.698094    2572 apiserver.go:52] "Watching apiserver"
Sep 10 15:42:08.711035 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.711010    2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 15:42:08.711384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.711362    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-tc4ns","kube-system/kube-apiserver-proxy-ip-10-0-134-36.ec2.internal","openshift-cluster-node-tuning-operator/tuned-plx44","openshift-dns/node-resolver-54n4f","openshift-network-diagnostics/network-check-target-jj8fc","openshift-network-operator/iptables-alerter-t67th","openshift-ovn-kubernetes/ovnkube-node-7sm72","kube-system/global-pull-secret-syncer-vz9z7","kube-system/konnectivity-agent-nf5db","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q","openshift-image-registry/node-ca-q944h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal","openshift-multus/multus-7zlxb","openshift-multus/multus-additional-cni-plugins-6zg8x"]
Sep 10 15:42:08.712887 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.712870    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.714182 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.714162    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.715228 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.715183    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 15:42:08.715228 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.715200    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 15:42:08.715228 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.715189    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 15:42:08.715422 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.715246    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 15:42:08.715422 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.715188    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 15:42:08.715517 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.715501    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.715752 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.715737    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mww2g\""
Sep 10 15:42:08.715980 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.715968    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 15:42:08.716139 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.716125    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-n7hmt\""
Sep 10 15:42:08.716180 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.716126    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 15:42:08.716221 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.716181    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 15:42:08.716296 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.716270    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 15:42:08.716680 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.716668    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-54n4f"
Sep 10 15:42:08.717032 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.717015    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 15:42:08.717753 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.717737    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:08.717827 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:08.717797    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:08.718063 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.718043    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:42:08.718157 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.718073    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 15:42:08.718388 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.718372    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zxhxk\""
Sep 10 15:42:08.718464 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.718374    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 15:42:08.718619 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.718606    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 15:42:08.718842 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.718824    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bg4rj\""
Sep 10 15:42:08.718949 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.718936    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.719630 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.719615    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 15:42:08.720185 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.720170    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nf5db"
Sep 10 15:42:08.720651 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.720636    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7cmh2\""
Sep 10 15:42:08.720737 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.720720    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 15:42:08.720774 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.720758    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 15:42:08.721348 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.721335    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:08.721435 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:08.721405    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:08.722038 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.722021    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 15:42:08.722109 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.722048    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9mvsp\""
Sep 10 15:42:08.722109 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.722052    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 15:42:08.723228 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.723212    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q944h"
Sep 10 15:42:08.724651 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.724615    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 15:37:07 +0000 UTC" deadline="2028-04-21 17:56:46.098547044 +0000 UTC"
Sep 10 15:42:08.724651 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.724648    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14138h14m37.373901196s"
Sep 10 15:42:08.724789 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.724718    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.725876 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.725668    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 15:42:08.725876 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.725666    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 15:42:08.726105 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.725669    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 15:42:08.726610 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.726370    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mzpl6\""
Sep 10 15:42:08.726610 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.726478    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:08.726610 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:08.726547    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:08.727186 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.727165    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-cnibin\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.727263 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.727210    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-multus-socket-dir-parent\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.727374 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.727245    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-sysctl-d\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.727440 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.727391    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8226797c-2abe-4e64-9da6-580fd99c29d7-tmp\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.727440 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.727405    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 15:42:08.727440 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.727427    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-cnibin\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.727573 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.727459    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.727573 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.727511    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-run-ovn\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.727573 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.727540    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-cni-netd\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.727751 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.727708    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-lib-modules\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.727751 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.727744    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-systemd-units\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.727842 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.727776    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-cni-bin\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.727842 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.727825    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.727938 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.727885    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.727990 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.727943    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-log-socket\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.727990 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.727973    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 15:42:08.727990 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.727976    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-sysconfig\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.728160 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.728032    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs\") pod \"network-metrics-daemon-tc4ns\" (UID: \"bde0d086-b64d-4188-b17c-4a69af69c9a5\") " pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:08.728160 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.728067    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r5qf\" (UniqueName: \"kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf\") pod \"network-check-target-jj8fc\" (UID: \"719ee58f-5767-4813-9b76-0ab62b2c8f3a\") " pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:08.728160 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.728107    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-slash\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.728160 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.728135    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-etc-openvswitch\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.728372 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.728242    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.728419 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.728290    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-system-cni-dir\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.728472 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.728457    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-host-var-lib-cni-bin\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.728530 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.728512    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t67th"
Sep 10 15:42:08.728577 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.728258    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 15:42:08.728621 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.728568    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b6cbt\""
Sep 10 15:42:08.728831 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.728516    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-etc-kubernetes\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.728896 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.728859    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-var-lib-openvswitch\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.728896 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.728887    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chxcj\" (UniqueName: \"kubernetes.io/projected/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-kube-api-access-chxcj\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.728981 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.728916    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-host-run-k8s-cni-cncf-io\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.728981 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.728944    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ca15df82-de7b-4bf8-a736-01c5537e53d2-multus-daemon-config\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.729068 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.728974    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-system-cni-dir\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.729115 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729011    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cccmh\" (UniqueName: \"kubernetes.io/projected/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-kube-api-access-cccmh\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.729161 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729131    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-run-netns\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.729329 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729205    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca15df82-de7b-4bf8-a736-01c5537e53d2-cni-binary-copy\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.729329 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729257    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-sysctl-conf\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.729447 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729326    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-var-lib-kubelet\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.729447 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729412    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p742n\" (UniqueName: \"kubernetes.io/projected/8226797c-2abe-4e64-9da6-580fd99c29d7-kube-api-access-p742n\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.729546 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729444    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1611fac1-d845-4d08-bdbf-72957655ab60-tmp-dir\") pod \"node-resolver-54n4f\" (UID: \"1611fac1-d845-4d08-bdbf-72957655ab60\") " pod="openshift-dns/node-resolver-54n4f"
Sep 10 15:42:08.729546 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729475    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.729546 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729504    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-modprobe-d\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.729683 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729546    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-systemd\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.729683 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729577    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7a6d6658-59b9-48e2-83e1-8a2b8750ecf7-agent-certs\") pod \"konnectivity-agent-nf5db\" (UID: \"7a6d6658-59b9-48e2-83e1-8a2b8750ecf7\") " pod="kube-system/konnectivity-agent-nf5db"
Sep 10 15:42:08.729683 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729605    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-hostroot\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.729683 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729628    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-ovnkube-script-lib\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.729683 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729657    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-host-var-lib-kubelet\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.729902 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729701    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-sys\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.729902 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729753    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-run-openvswitch\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.729902 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729782    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-ovnkube-config\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.729902 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729810    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-multus-conf-dir\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.729902 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729828    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-host-run-multus-certs\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.729902 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729850    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-run-systemd\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.730158 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.729878    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-env-overrides\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.730158 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.730011    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-host-var-lib-cni-multus\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.730158 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.730040    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s4r8\" (UniqueName: \"kubernetes.io/projected/ca15df82-de7b-4bf8-a736-01c5537e53d2-kube-api-access-7s4r8\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.730158 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.730065    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-run\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.730158 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.730091    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1611fac1-d845-4d08-bdbf-72957655ab60-hosts-file\") pod \"node-resolver-54n4f\" (UID: \"1611fac1-d845-4d08-bdbf-72957655ab60\") " pod="openshift-dns/node-resolver-54n4f"
Sep 10 15:42:08.730158 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.730118    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-multus-cni-dir\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.730445 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.730147    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-host\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.730445 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.730242    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-tuned\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.730445 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.730280    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7a6d6658-59b9-48e2-83e1-8a2b8750ecf7-konnectivity-ca\") pod \"konnectivity-agent-nf5db\" (UID: \"7a6d6658-59b9-48e2-83e1-8a2b8750ecf7\") " pod="kube-system/konnectivity-agent-nf5db"
Sep 10 15:42:08.730445 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.730330    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-host-run-netns\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.730445 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.730359    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.730445 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.730385    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-ovn-node-metrics-cert\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.730445 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.730412    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-kubernetes\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.730445 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.730437    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzmq6\" (UniqueName: \"kubernetes.io/projected/1611fac1-d845-4d08-bdbf-72957655ab60-kube-api-access-kzmq6\") pod \"node-resolver-54n4f\" (UID: \"1611fac1-d845-4d08-bdbf-72957655ab60\") " pod="openshift-dns/node-resolver-54n4f"
Sep 10 15:42:08.730744 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.730464    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-kubelet\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.730744 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.730571    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f6j5\" (UniqueName: \"kubernetes.io/projected/bde0d086-b64d-4188-b17c-4a69af69c9a5-kube-api-access-4f6j5\") pod \"network-metrics-daemon-tc4ns\" (UID: \"bde0d086-b64d-4188-b17c-4a69af69c9a5\") " pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:08.730744 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.730635    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-node-log\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.731266 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.731220    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4d5k2\""
Sep 10 15:42:08.731475 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.731452    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:42:08.731565 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.731452    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 15:42:08.731565 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.731489    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 15:42:08.732781 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.732763    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 15:42:08.761257 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.761227    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g4gsn"
Sep 10 15:42:08.771098 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.771078    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g4gsn"
Sep 10 15:42:08.801815 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.801790    2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:42:08.822177 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.822152    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 15:42:08.831134 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831112    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-lib-modules\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.831244 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831142    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9701fbb2-7db9-40d3-af55-f611b396f4cd-registration-dir\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.831244 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831163    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-systemd-units\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.831244 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831186    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-cni-bin\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.831244 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831228    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.831618 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831256    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.831618 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831260    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-systemd-units\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.831618 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831283    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-lib-modules\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.831618 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831293    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-cni-bin\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.831618 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831354    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-log-socket\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.831618 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831415    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-sysconfig\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.831618 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831443    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-log-socket\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.831618 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831454    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs\") pod \"network-metrics-daemon-tc4ns\" (UID: \"bde0d086-b64d-4188-b17c-4a69af69c9a5\") " pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:08.831618 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831472    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.831618 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831487    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-sysconfig\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.831618 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831511    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r5qf\" (UniqueName: \"kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf\") pod \"network-check-target-jj8fc\" (UID: \"719ee58f-5767-4813-9b76-0ab62b2c8f3a\") " pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:08.831618 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831553    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-slash\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.831618 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831583    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-etc-openvswitch\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.831618 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:08.831586    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:08.831618 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831616    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.832269 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831662    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.832269 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831671    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-slash\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.832269 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831744    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-etc-openvswitch\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.832269 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:08.831878    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs podName:bde0d086-b64d-4188-b17c-4a69af69c9a5 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:09.331634704 +0000 UTC m=+2.004044779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs") pod "network-metrics-daemon-tc4ns" (UID: "bde0d086-b64d-4188-b17c-4a69af69c9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:08.832269 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831901    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-system-cni-dir\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.832269 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831925    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-host-var-lib-cni-bin\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.832269 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831965    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-etc-kubernetes\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.832269 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831990    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-var-lib-openvswitch\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.832269 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831992    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-system-cni-dir\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.832269 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.832009    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-host-var-lib-cni-bin\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.832269 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.831987    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.832269 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.832070    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-var-lib-openvswitch\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.832269 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.832040    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-etc-kubernetes\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.832269 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.832025    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chxcj\" (UniqueName: \"kubernetes.io/projected/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-kube-api-access-chxcj\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.832996 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.832624    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-host-run-k8s-cni-cncf-io\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.832996 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.832681    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ca15df82-de7b-4bf8-a736-01c5537e53d2-multus-daemon-config\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.832996 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.832716    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-system-cni-dir\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.832996 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.832741    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cccmh\" (UniqueName: \"kubernetes.io/projected/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-kube-api-access-cccmh\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.832996 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.832777    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f221b59a-f955-4ce1-a9b4-e4efe942a24a-serviceca\") pod \"node-ca-q944h\" (UID: \"f221b59a-f955-4ce1-a9b4-e4efe942a24a\") " pod="openshift-image-registry/node-ca-q944h"
Sep 10 15:42:08.832996 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.832816    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-host-run-k8s-cni-cncf-io\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.832996 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.832829    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drvp6\" (UniqueName: \"kubernetes.io/projected/f221b59a-f955-4ce1-a9b4-e4efe942a24a-kube-api-access-drvp6\") pod \"node-ca-q944h\" (UID: \"f221b59a-f955-4ce1-a9b4-e4efe942a24a\") " pod="openshift-image-registry/node-ca-q944h"
Sep 10 15:42:08.832996 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.832858    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-run-netns\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.832996 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.832893    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-system-cni-dir\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.833413 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.833343    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca15df82-de7b-4bf8-a736-01c5537e53d2-cni-binary-copy\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.833413 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.833345    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-run-netns\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.833413 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.833389    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-sysctl-conf\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.833550 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.833446    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-var-lib-kubelet\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.833550 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.833494    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p742n\" (UniqueName: \"kubernetes.io/projected/8226797c-2abe-4e64-9da6-580fd99c29d7-kube-api-access-p742n\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.833550 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.833527    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1611fac1-d845-4d08-bdbf-72957655ab60-tmp-dir\") pod \"node-resolver-54n4f\" (UID: \"1611fac1-d845-4d08-bdbf-72957655ab60\") " pod="openshift-dns/node-resolver-54n4f"
Sep 10 15:42:08.833690 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.833555    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.833690 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.833583    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-sysctl-conf\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.833690 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.833630    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.833690 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.833669    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-modprobe-d\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.833871 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.833699    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-var-lib-kubelet\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.833871 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.833698    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-systemd\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.833871 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.833746    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9701fbb2-7db9-40d3-af55-f611b396f4cd-device-dir\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.833871 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.833749    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-systemd\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.833871 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.833548    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca15df82-de7b-4bf8-a736-01c5537e53d2-cni-binary-copy\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.833871 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.833838    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-modprobe-d\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.834139 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.833878    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9701fbb2-7db9-40d3-af55-f611b396f4cd-sys-fs\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.834139 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.833919    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7a6d6658-59b9-48e2-83e1-8a2b8750ecf7-agent-certs\") pod \"konnectivity-agent-nf5db\" (UID: \"7a6d6658-59b9-48e2-83e1-8a2b8750ecf7\") " pod="kube-system/konnectivity-agent-nf5db"
Sep 10 15:42:08.834139 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.833987    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-hostroot\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.834139 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834057    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1611fac1-d845-4d08-bdbf-72957655ab60-tmp-dir\") pod \"node-resolver-54n4f\" (UID: \"1611fac1-d845-4d08-bdbf-72957655ab60\") " pod="openshift-dns/node-resolver-54n4f"
Sep 10 15:42:08.834139 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834076    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-hostroot\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.834139 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834097    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9701fbb2-7db9-40d3-af55-f611b396f4cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.834418 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834154    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret\") pod \"global-pull-secret-syncer-vz9z7\" (UID: \"a7d7c20f-a31c-4468-a822-a7e2601fe3e4\") " pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:08.834418 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834218    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-ovnkube-script-lib\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.834418 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834246    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-host-var-lib-kubelet\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.834418 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834276    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-sys\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.834418 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834332    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9701fbb2-7db9-40d3-af55-f611b396f4cd-socket-dir\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.834418 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834388    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-run-openvswitch\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.834684 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834414    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-ovnkube-config\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.834684 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834467    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-multus-conf-dir\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.834684 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834503    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-host-run-multus-certs\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.834684 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834558    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcdxk\" (UniqueName: \"kubernetes.io/projected/9701fbb2-7db9-40d3-af55-f611b396f4cd-kube-api-access-kcdxk\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.834684 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834572    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ca15df82-de7b-4bf8-a736-01c5537e53d2-multus-daemon-config\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.834684 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834597    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-run-systemd\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.834684 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834657    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-run-systemd\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.834980 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834730    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-multus-conf-dir\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.834980 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834767    2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 15:42:08.834980 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834887    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-sys\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.834980 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834932    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-env-overrides\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.835156 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.834999    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-host-var-lib-cni-multus\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.835156 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.835083    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s4r8\" (UniqueName: \"kubernetes.io/projected/ca15df82-de7b-4bf8-a736-01c5537e53d2-kube-api-access-7s4r8\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.835248 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.835159    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-run\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.835248 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.835187    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9701fbb2-7db9-40d3-af55-f611b396f4cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.835248 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.835219    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2kn2\" (UniqueName: \"kubernetes.io/projected/8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded-kube-api-access-t2kn2\") pod \"iptables-alerter-t67th\" (UID: \"8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded\") " pod="openshift-network-operator/iptables-alerter-t67th"
Sep 10 15:42:08.836646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.835409    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1611fac1-d845-4d08-bdbf-72957655ab60-hosts-file\") pod \"node-resolver-54n4f\" (UID: \"1611fac1-d845-4d08-bdbf-72957655ab60\") " pod="openshift-dns/node-resolver-54n4f"
Sep 10 15:42:08.836646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.835426    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-host-run-multus-certs\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.836646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.835411    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-ovnkube-config\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.836646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.835482    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-host-var-lib-cni-multus\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.836646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.835482    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1611fac1-d845-4d08-bdbf-72957655ab60-hosts-file\") pod \"node-resolver-54n4f\" (UID: \"1611fac1-d845-4d08-bdbf-72957655ab60\") " pod="openshift-dns/node-resolver-54n4f"
Sep 10 15:42:08.836646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.835548    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-host-var-lib-kubelet\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.836646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.835570    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-multus-cni-dir\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.836646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.835597    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-run-openvswitch\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.836646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.835617    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-host\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.836646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.835648    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-tuned\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.836646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.835735    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-kubelet-config\") pod \"global-pull-secret-syncer-vz9z7\" (UID: \"a7d7c20f-a31c-4468-a822-a7e2601fe3e4\") " pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:08.836646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.835800    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-run\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.836646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.835905    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-multus-cni-dir\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.836646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.835972    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-host\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.836646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.836164    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-env-overrides\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.836646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.836221    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-ovnkube-script-lib\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.836646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.836247    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded-host-slash\") pod \"iptables-alerter-t67th\" (UID: \"8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded\") " pod="openshift-network-operator/iptables-alerter-t67th"
Sep 10 15:42:08.836646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.836293    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7a6d6658-59b9-48e2-83e1-8a2b8750ecf7-konnectivity-ca\") pod \"konnectivity-agent-nf5db\" (UID: \"7a6d6658-59b9-48e2-83e1-8a2b8750ecf7\") " pod="kube-system/konnectivity-agent-nf5db"
Sep 10 15:42:08.837514 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.836363    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-host-run-netns\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.837514 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.836396    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.837514 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.836430    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-ovn-node-metrics-cert\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.837514 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.836458    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-kubernetes\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.837514 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.836487    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-dbus\") pod \"global-pull-secret-syncer-vz9z7\" (UID: \"a7d7c20f-a31c-4468-a822-a7e2601fe3e4\") " pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:08.837514 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.836534    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded-iptables-alerter-script\") pod \"iptables-alerter-t67th\" (UID: \"8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded\") " pod="openshift-network-operator/iptables-alerter-t67th"
Sep 10 15:42:08.837514 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.836582    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzmq6\" (UniqueName: \"kubernetes.io/projected/1611fac1-d845-4d08-bdbf-72957655ab60-kube-api-access-kzmq6\") pod \"node-resolver-54n4f\" (UID: \"1611fac1-d845-4d08-bdbf-72957655ab60\") " pod="openshift-dns/node-resolver-54n4f"
Sep 10 15:42:08.837514 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.836622    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-kubelet\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.837514 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.836669    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f6j5\" (UniqueName: \"kubernetes.io/projected/bde0d086-b64d-4188-b17c-4a69af69c9a5-kube-api-access-4f6j5\") pod \"network-metrics-daemon-tc4ns\" (UID: \"bde0d086-b64d-4188-b17c-4a69af69c9a5\") " pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:08.837514 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.836702    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-node-log\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.837514 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.836865    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-cnibin\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.837514 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.836899    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-multus-socket-dir-parent\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.837514 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.836944    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7a6d6658-59b9-48e2-83e1-8a2b8750ecf7-konnectivity-ca\") pod \"konnectivity-agent-nf5db\" (UID: \"7a6d6658-59b9-48e2-83e1-8a2b8750ecf7\") " pod="kube-system/konnectivity-agent-nf5db"
Sep 10 15:42:08.837514 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.836953    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-sysctl-d\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.837514 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.836981    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8226797c-2abe-4e64-9da6-580fd99c29d7-tmp\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.837514 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.837022    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-cnibin\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.837514 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.837054    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.838262 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.837090    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f221b59a-f955-4ce1-a9b4-e4efe942a24a-host\") pod \"node-ca-q944h\" (UID: \"f221b59a-f955-4ce1-a9b4-e4efe942a24a\") " pod="openshift-image-registry/node-ca-q944h"
Sep 10 15:42:08.838262 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.837116    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-run-ovn\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.838262 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.837145    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-cni-netd\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.838262 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.837258    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-cni-netd\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.838262 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.837270    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-host-run-netns\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.838262 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.837351    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-kubernetes\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.838262 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.837568    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-node-log\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.838262 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.837632    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-cnibin\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.838262 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.837678    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-cnibin\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.838262 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.837760    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ca15df82-de7b-4bf8-a736-01c5537e53d2-multus-socket-dir-parent\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.838262 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.837812    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.838262 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.837810    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-run-ovn\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.838262 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.837940    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-host-kubelet\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.838262 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.838144    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-sysctl-d\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.838938 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.838435    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.839470 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:08.839446    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:08.839470 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:08.839465    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:08.839616 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:08.839495    2572 projected.go:196] Error preparing data for projected volume kube-api-access-5r5qf for pod openshift-network-diagnostics/network-check-target-jj8fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:08.839616 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:08.839601    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf podName:719ee58f-5767-4813-9b76-0ab62b2c8f3a nodeName:}" failed. No retries permitted until 2026-09-10 15:42:09.339570823 +0000 UTC m=+2.011980904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5r5qf" (UniqueName: "kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf") pod "network-check-target-jj8fc" (UID: "719ee58f-5767-4813-9b76-0ab62b2c8f3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:08.840691 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.840652    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8226797c-2abe-4e64-9da6-580fd99c29d7-etc-tuned\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.840840 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.840825    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7a6d6658-59b9-48e2-83e1-8a2b8750ecf7-agent-certs\") pod \"konnectivity-agent-nf5db\" (UID: \"7a6d6658-59b9-48e2-83e1-8a2b8750ecf7\") " pod="kube-system/konnectivity-agent-nf5db"
Sep 10 15:42:08.840908 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.840878    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-ovn-node-metrics-cert\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.842350 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.842291    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8226797c-2abe-4e64-9da6-580fd99c29d7-tmp\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.843870 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.843843    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chxcj\" (UniqueName: \"kubernetes.io/projected/8ddd4dc1-1122-41e0-a54a-67fb8008d7b9-kube-api-access-chxcj\") pod \"ovnkube-node-7sm72\" (UID: \"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:08.844111 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.844093    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s4r8\" (UniqueName: \"kubernetes.io/projected/ca15df82-de7b-4bf8-a736-01c5537e53d2-kube-api-access-7s4r8\") pod \"multus-7zlxb\" (UID: \"ca15df82-de7b-4bf8-a736-01c5537e53d2\") " pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:08.844274 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.844247    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cccmh\" (UniqueName: \"kubernetes.io/projected/eacf5dc6-da67-4d9e-b76c-8a2e946bf637-kube-api-access-cccmh\") pod \"multus-additional-cni-plugins-6zg8x\" (UID: \"eacf5dc6-da67-4d9e-b76c-8a2e946bf637\") " pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:08.845143 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.845124    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f6j5\" (UniqueName: \"kubernetes.io/projected/bde0d086-b64d-4188-b17c-4a69af69c9a5-kube-api-access-4f6j5\") pod \"network-metrics-daemon-tc4ns\" (UID: \"bde0d086-b64d-4188-b17c-4a69af69c9a5\") " pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:08.845451 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.845432    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzmq6\" (UniqueName: \"kubernetes.io/projected/1611fac1-d845-4d08-bdbf-72957655ab60-kube-api-access-kzmq6\") pod \"node-resolver-54n4f\" (UID: \"1611fac1-d845-4d08-bdbf-72957655ab60\") " pod="openshift-dns/node-resolver-54n4f"
Sep 10 15:42:08.846016 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.846000    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p742n\" (UniqueName: \"kubernetes.io/projected/8226797c-2abe-4e64-9da6-580fd99c29d7-kube-api-access-p742n\") pod \"tuned-plx44\" (UID: \"8226797c-2abe-4e64-9da6-580fd99c29d7\") " pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:08.932094 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:08.932055    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87d7a97be63e6b3b27b586fea74839e.slice/crio-38a711f4c35281f773d9bf7044f6077fa7c661877a68a0d2f7926c867db660f4 WatchSource:0}: Error finding container 38a711f4c35281f773d9bf7044f6077fa7c661877a68a0d2f7926c867db660f4: Status 404 returned error can't find the container with id 38a711f4c35281f773d9bf7044f6077fa7c661877a68a0d2f7926c867db660f4
Sep 10 15:42:08.935948 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.935931    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:42:08.937726 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.937705    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f221b59a-f955-4ce1-a9b4-e4efe942a24a-serviceca\") pod \"node-ca-q944h\" (UID: \"f221b59a-f955-4ce1-a9b4-e4efe942a24a\") " pod="openshift-image-registry/node-ca-q944h"
Sep 10 15:42:08.937808 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.937737    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drvp6\" (UniqueName: \"kubernetes.io/projected/f221b59a-f955-4ce1-a9b4-e4efe942a24a-kube-api-access-drvp6\") pod \"node-ca-q944h\" (UID: \"f221b59a-f955-4ce1-a9b4-e4efe942a24a\") " pod="openshift-image-registry/node-ca-q944h"
Sep 10 15:42:08.937808 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.937766    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9701fbb2-7db9-40d3-af55-f611b396f4cd-device-dir\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.937808 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.937792    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9701fbb2-7db9-40d3-af55-f611b396f4cd-sys-fs\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.937971 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.937815    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9701fbb2-7db9-40d3-af55-f611b396f4cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.937971 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.937854    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret\") pod \"global-pull-secret-syncer-vz9z7\" (UID: \"a7d7c20f-a31c-4468-a822-a7e2601fe3e4\") " pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:08.937971 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.937880    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9701fbb2-7db9-40d3-af55-f611b396f4cd-socket-dir\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.937971 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.937905    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9701fbb2-7db9-40d3-af55-f611b396f4cd-device-dir\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.937971 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.937923    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcdxk\" (UniqueName: \"kubernetes.io/projected/9701fbb2-7db9-40d3-af55-f611b396f4cd-kube-api-access-kcdxk\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.937971 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.937925    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9701fbb2-7db9-40d3-af55-f611b396f4cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.937971 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.937927    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9701fbb2-7db9-40d3-af55-f611b396f4cd-sys-fs\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.938362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.937962    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9701fbb2-7db9-40d3-af55-f611b396f4cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.938362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.938007    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2kn2\" (UniqueName: \"kubernetes.io/projected/8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded-kube-api-access-t2kn2\") pod \"iptables-alerter-t67th\" (UID: \"8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded\") " pod="openshift-network-operator/iptables-alerter-t67th"
Sep 10 15:42:08.938362 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:08.938015    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:08.938362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.938035    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-kubelet-config\") pod \"global-pull-secret-syncer-vz9z7\" (UID: \"a7d7c20f-a31c-4468-a822-a7e2601fe3e4\") " pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:08.938362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.938058    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded-host-slash\") pod \"iptables-alerter-t67th\" (UID: \"8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded\") " pod="openshift-network-operator/iptables-alerter-t67th"
Sep 10 15:42:08.938362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.938067    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9701fbb2-7db9-40d3-af55-f611b396f4cd-socket-dir\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.938362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.938092    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-dbus\") pod \"global-pull-secret-syncer-vz9z7\" (UID: \"a7d7c20f-a31c-4468-a822-a7e2601fe3e4\") " pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:08.938362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.938104    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9701fbb2-7db9-40d3-af55-f611b396f4cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.938362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.938123    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded-iptables-alerter-script\") pod \"iptables-alerter-t67th\" (UID: \"8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded\") " pod="openshift-network-operator/iptables-alerter-t67th"
Sep 10 15:42:08.938362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.938140    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded-host-slash\") pod \"iptables-alerter-t67th\" (UID: \"8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded\") " pod="openshift-network-operator/iptables-alerter-t67th"
Sep 10 15:42:08.938362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.938125    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-kubelet-config\") pod \"global-pull-secret-syncer-vz9z7\" (UID: \"a7d7c20f-a31c-4468-a822-a7e2601fe3e4\") " pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:08.938362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.938163    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f221b59a-f955-4ce1-a9b4-e4efe942a24a-host\") pod \"node-ca-q944h\" (UID: \"f221b59a-f955-4ce1-a9b4-e4efe942a24a\") " pod="openshift-image-registry/node-ca-q944h"
Sep 10 15:42:08.938362 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:08.938207    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret podName:a7d7c20f-a31c-4468-a822-a7e2601fe3e4 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:09.438186334 +0000 UTC m=+2.110596406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret") pod "global-pull-secret-syncer-vz9z7" (UID: "a7d7c20f-a31c-4468-a822-a7e2601fe3e4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:08.938362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.938203    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f221b59a-f955-4ce1-a9b4-e4efe942a24a-serviceca\") pod \"node-ca-q944h\" (UID: \"f221b59a-f955-4ce1-a9b4-e4efe942a24a\") " pod="openshift-image-registry/node-ca-q944h"
Sep 10 15:42:08.938362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.938213    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-dbus\") pod \"global-pull-secret-syncer-vz9z7\" (UID: \"a7d7c20f-a31c-4468-a822-a7e2601fe3e4\") " pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:08.938362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.938244    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9701fbb2-7db9-40d3-af55-f611b396f4cd-registration-dir\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.938362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.938271    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f221b59a-f955-4ce1-a9b4-e4efe942a24a-host\") pod \"node-ca-q944h\" (UID: \"f221b59a-f955-4ce1-a9b4-e4efe942a24a\") " pod="openshift-image-registry/node-ca-q944h"
Sep 10 15:42:08.938864 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.938333    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9701fbb2-7db9-40d3-af55-f611b396f4cd-registration-dir\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.938864 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.938733    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded-iptables-alerter-script\") pod \"iptables-alerter-t67th\" (UID: \"8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded\") " pod="openshift-network-operator/iptables-alerter-t67th"
Sep 10 15:42:08.948021 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.947997    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2kn2\" (UniqueName: \"kubernetes.io/projected/8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded-kube-api-access-t2kn2\") pod \"iptables-alerter-t67th\" (UID: \"8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded\") " pod="openshift-network-operator/iptables-alerter-t67th"
Sep 10 15:42:08.948124 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.947999    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcdxk\" (UniqueName: \"kubernetes.io/projected/9701fbb2-7db9-40d3-af55-f611b396f4cd-kube-api-access-kcdxk\") pod \"aws-ebs-csi-driver-node-wb59q\" (UID: \"9701fbb2-7db9-40d3-af55-f611b396f4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:08.948124 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.948074    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drvp6\" (UniqueName: \"kubernetes.io/projected/f221b59a-f955-4ce1-a9b4-e4efe942a24a-kube-api-access-drvp6\") pod \"node-ca-q944h\" (UID: \"f221b59a-f955-4ce1-a9b4-e4efe942a24a\") " pod="openshift-image-registry/node-ca-q944h"
Sep 10 15:42:08.989362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:08.989317    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal" event={"ID":"f87d7a97be63e6b3b27b586fea74839e","Type":"ContainerStarted","Data":"38a711f4c35281f773d9bf7044f6077fa7c661877a68a0d2f7926c867db660f4"}
Sep 10 15:42:09.038636 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:09.038576    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:09.043845 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:09.043821    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7zlxb"
Sep 10 15:42:09.046356 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:09.046332    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ddd4dc1_1122_41e0_a54a_67fb8008d7b9.slice/crio-0af8d40f059c11b9ad83fa01f338ff0bc2dac4142dbe6f325519418d4716f246 WatchSource:0}: Error finding container 0af8d40f059c11b9ad83fa01f338ff0bc2dac4142dbe6f325519418d4716f246: Status 404 returned error can't find the container with id 0af8d40f059c11b9ad83fa01f338ff0bc2dac4142dbe6f325519418d4716f246
Sep 10 15:42:09.051918 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:09.051902    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca15df82_de7b_4bf8_a736_01c5537e53d2.slice/crio-51a0105c7a6183519f16207024fe4798beeda390167b72db5536f2c6e34eadc6 WatchSource:0}: Error finding container 51a0105c7a6183519f16207024fe4798beeda390167b72db5536f2c6e34eadc6: Status 404 returned error can't find the container with id 51a0105c7a6183519f16207024fe4798beeda390167b72db5536f2c6e34eadc6
Sep 10 15:42:09.066258 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:09.066239    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-plx44"
Sep 10 15:42:09.073009 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:09.072990    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8226797c_2abe_4e64_9da6_580fd99c29d7.slice/crio-89351bc680f52d8cf871c1532baa42cda4b98b7ae7ba0dd6add5e2944ab0f74e WatchSource:0}: Error finding container 89351bc680f52d8cf871c1532baa42cda4b98b7ae7ba0dd6add5e2944ab0f74e: Status 404 returned error can't find the container with id 89351bc680f52d8cf871c1532baa42cda4b98b7ae7ba0dd6add5e2944ab0f74e
Sep 10 15:42:09.079060 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:09.079042    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-54n4f"
Sep 10 15:42:09.085898 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:09.085871    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1611fac1_d845_4d08_bdbf_72957655ab60.slice/crio-1ccd76a866442c634037acfefdf9568cda02ec466041e851c5798459a23f9d0f WatchSource:0}: Error finding container 1ccd76a866442c634037acfefdf9568cda02ec466041e851c5798459a23f9d0f: Status 404 returned error can't find the container with id 1ccd76a866442c634037acfefdf9568cda02ec466041e851c5798459a23f9d0f
Sep 10 15:42:09.094193 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:09.094174    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6zg8x"
Sep 10 15:42:09.098002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:09.097985    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nf5db"
Sep 10 15:42:09.101278 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:09.101258    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeacf5dc6_da67_4d9e_b76c_8a2e946bf637.slice/crio-c8a31e0bf823fba5afbfe08de883a1b1226c7387f16b7ba78f94172af14cf0cc WatchSource:0}: Error finding container c8a31e0bf823fba5afbfe08de883a1b1226c7387f16b7ba78f94172af14cf0cc: Status 404 returned error can't find the container with id c8a31e0bf823fba5afbfe08de883a1b1226c7387f16b7ba78f94172af14cf0cc
Sep 10 15:42:09.107324 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:09.107288    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6d6658_59b9_48e2_83e1_8a2b8750ecf7.slice/crio-a4eecd52a12a0b23d300c3ccc9693ad04f6d76f583b82625ea53eb0ab14d554f WatchSource:0}: Error finding container a4eecd52a12a0b23d300c3ccc9693ad04f6d76f583b82625ea53eb0ab14d554f: Status 404 returned error can't find the container with id a4eecd52a12a0b23d300c3ccc9693ad04f6d76f583b82625ea53eb0ab14d554f
Sep 10 15:42:09.114605 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:09.114586    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q944h"
Sep 10 15:42:09.121295 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:09.121274    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf221b59a_f955_4ce1_a9b4_e4efe942a24a.slice/crio-b915702c6f6af028cedda5271245241e1384a68a7a67661be8425736af9fba72 WatchSource:0}: Error finding container b915702c6f6af028cedda5271245241e1384a68a7a67661be8425736af9fba72: Status 404 returned error can't find the container with id b915702c6f6af028cedda5271245241e1384a68a7a67661be8425736af9fba72
Sep 10 15:42:09.129399 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:09.129380    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q"
Sep 10 15:42:09.135860 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:09.135842    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9701fbb2_7db9_40d3_af55_f611b396f4cd.slice/crio-cb13413032c227b911d0f9151be19795fad7d0509fb6199fa026b97621a57c7f WatchSource:0}: Error finding container cb13413032c227b911d0f9151be19795fad7d0509fb6199fa026b97621a57c7f: Status 404 returned error can't find the container with id cb13413032c227b911d0f9151be19795fad7d0509fb6199fa026b97621a57c7f
Sep 10 15:42:09.138805 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:09.138791    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t67th"
Sep 10 15:42:09.145335 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:09.145315    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7033d6_0cdc_4ef5_8f25_9fc1b4b7aded.slice/crio-53fb1ee7d1bf0aecc10b14b4f746964cd20abb79bc69f5b555e219ef709676e7 WatchSource:0}: Error finding container 53fb1ee7d1bf0aecc10b14b4f746964cd20abb79bc69f5b555e219ef709676e7: Status 404 returned error can't find the container with id 53fb1ee7d1bf0aecc10b14b4f746964cd20abb79bc69f5b555e219ef709676e7
Sep 10 15:42:09.281449 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:09.281422    2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:42:09.340528 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:09.340501    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs\") pod \"network-metrics-daemon-tc4ns\" (UID: \"bde0d086-b64d-4188-b17c-4a69af69c9a5\") " pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:09.340681 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:09.340533    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r5qf\" (UniqueName: \"kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf\") pod \"network-check-target-jj8fc\" (UID: \"719ee58f-5767-4813-9b76-0ab62b2c8f3a\") " pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:09.340681 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:09.340635    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:09.340681 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:09.340646    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:09.340681 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:09.340660    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:09.340681 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:09.340671    2572 projected.go:196] Error preparing data for projected volume kube-api-access-5r5qf for pod openshift-network-diagnostics/network-check-target-jj8fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:09.340837 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:09.340706    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs podName:bde0d086-b64d-4188-b17c-4a69af69c9a5 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:10.340692918 +0000 UTC m=+3.013102979 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs") pod "network-metrics-daemon-tc4ns" (UID: "bde0d086-b64d-4188-b17c-4a69af69c9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:09.340837 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:09.340723    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf podName:719ee58f-5767-4813-9b76-0ab62b2c8f3a nodeName:}" failed. No retries permitted until 2026-09-10 15:42:10.340715641 +0000 UTC m=+3.013125701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5r5qf" (UniqueName: "kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf") pod "network-check-target-jj8fc" (UID: "719ee58f-5767-4813-9b76-0ab62b2c8f3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:09.441486 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:09.441450    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret\") pod \"global-pull-secret-syncer-vz9z7\" (UID: \"a7d7c20f-a31c-4468-a822-a7e2601fe3e4\") " pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:09.441661 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:09.441588    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:09.441741 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:09.441665    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret podName:a7d7c20f-a31c-4468-a822-a7e2601fe3e4 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:10.441642962 +0000 UTC m=+3.114053045 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret") pod "global-pull-secret-syncer-vz9z7" (UID: "a7d7c20f-a31c-4468-a822-a7e2601fe3e4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:09.772990 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:09.772890    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:37:08 +0000 UTC" deadline="2028-05-27 16:51:38.398523038 +0000 UTC"
Sep 10 15:42:09.772990 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:09.772932    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15001h9m28.625595105s"
Sep 10 15:42:09.999600 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:09.999543    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q944h" event={"ID":"f221b59a-f955-4ce1-a9b4-e4efe942a24a","Type":"ContainerStarted","Data":"b915702c6f6af028cedda5271245241e1384a68a7a67661be8425736af9fba72"}
Sep 10 15:42:10.007408 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:10.007332    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-54n4f" event={"ID":"1611fac1-d845-4d08-bdbf-72957655ab60","Type":"ContainerStarted","Data":"1ccd76a866442c634037acfefdf9568cda02ec466041e851c5798459a23f9d0f"}
Sep 10 15:42:10.020448 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:10.020407    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7zlxb" event={"ID":"ca15df82-de7b-4bf8-a736-01c5537e53d2","Type":"ContainerStarted","Data":"51a0105c7a6183519f16207024fe4798beeda390167b72db5536f2c6e34eadc6"}
Sep 10 15:42:10.026024 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:10.025942    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t67th" event={"ID":"8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded","Type":"ContainerStarted","Data":"53fb1ee7d1bf0aecc10b14b4f746964cd20abb79bc69f5b555e219ef709676e7"}
Sep 10 15:42:10.029875 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:10.029844    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q" event={"ID":"9701fbb2-7db9-40d3-af55-f611b396f4cd","Type":"ContainerStarted","Data":"cb13413032c227b911d0f9151be19795fad7d0509fb6199fa026b97621a57c7f"}
Sep 10 15:42:10.033786 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:10.033756    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nf5db" event={"ID":"7a6d6658-59b9-48e2-83e1-8a2b8750ecf7","Type":"ContainerStarted","Data":"a4eecd52a12a0b23d300c3ccc9693ad04f6d76f583b82625ea53eb0ab14d554f"}
Sep 10 15:42:10.041840 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:10.041810    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zg8x" event={"ID":"eacf5dc6-da67-4d9e-b76c-8a2e946bf637","Type":"ContainerStarted","Data":"c8a31e0bf823fba5afbfe08de883a1b1226c7387f16b7ba78f94172af14cf0cc"}
Sep 10 15:42:10.048617 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:10.048368    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-plx44" event={"ID":"8226797c-2abe-4e64-9da6-580fd99c29d7","Type":"ContainerStarted","Data":"89351bc680f52d8cf871c1532baa42cda4b98b7ae7ba0dd6add5e2944ab0f74e"}
Sep 10 15:42:10.052762 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:10.052700    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7sm72" event={"ID":"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9","Type":"ContainerStarted","Data":"0af8d40f059c11b9ad83fa01f338ff0bc2dac4142dbe6f325519418d4716f246"}
Sep 10 15:42:10.136864 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:10.136826    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872387aaa69ba3ec01d1d25ea3dd5b1d.slice/crio-89d1872498e52c623250c0e179f7f305e7f38fc3609c96c3ef7041ca703326f4 WatchSource:0}: Error finding container 89d1872498e52c623250c0e179f7f305e7f38fc3609c96c3ef7041ca703326f4: Status 404 returned error can't find the container with id 89d1872498e52c623250c0e179f7f305e7f38fc3609c96c3ef7041ca703326f4
Sep 10 15:42:10.219709 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:10.219408    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:42:10.349769 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:10.348742    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r5qf\" (UniqueName: \"kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf\") pod \"network-check-target-jj8fc\" (UID: \"719ee58f-5767-4813-9b76-0ab62b2c8f3a\") " pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:10.349769 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:10.348864    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs\") pod \"network-metrics-daemon-tc4ns\" (UID: \"bde0d086-b64d-4188-b17c-4a69af69c9a5\") " pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:10.349769 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:10.349056    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:10.349769 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:10.349129    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs podName:bde0d086-b64d-4188-b17c-4a69af69c9a5 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:12.349108953 +0000 UTC m=+5.021519017 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs") pod "network-metrics-daemon-tc4ns" (UID: "bde0d086-b64d-4188-b17c-4a69af69c9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:10.349769 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:10.349579    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:10.349769 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:10.349599    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:10.349769 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:10.349611    2572 projected.go:196] Error preparing data for projected volume kube-api-access-5r5qf for pod openshift-network-diagnostics/network-check-target-jj8fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:10.349769 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:10.349662    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf podName:719ee58f-5767-4813-9b76-0ab62b2c8f3a nodeName:}" failed. No retries permitted until 2026-09-10 15:42:12.349645971 +0000 UTC m=+5.022056051 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5r5qf" (UniqueName: "kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf") pod "network-check-target-jj8fc" (UID: "719ee58f-5767-4813-9b76-0ab62b2c8f3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:10.450481 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:10.449861    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret\") pod \"global-pull-secret-syncer-vz9z7\" (UID: \"a7d7c20f-a31c-4468-a822-a7e2601fe3e4\") " pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:10.450481 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:10.450011    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:10.450481 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:10.450070    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret podName:a7d7c20f-a31c-4468-a822-a7e2601fe3e4 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:12.450050755 +0000 UTC m=+5.122460834 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret") pod "global-pull-secret-syncer-vz9z7" (UID: "a7d7c20f-a31c-4468-a822-a7e2601fe3e4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:10.773751 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:10.773565    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:37:08 +0000 UTC" deadline="2028-06-07 05:26:18.84986853 +0000 UTC"
Sep 10 15:42:10.773751 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:10.773626    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15253h44m8.076246733s"
Sep 10 15:42:10.986210 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:10.986174    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:10.986406 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:10.986326    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:10.986765 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:10.986745    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:10.986870 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:10.986854    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:10.986936 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:10.986922    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:10.987033 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:10.986995    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:11.059080 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:11.058761    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-36.ec2.internal" event={"ID":"872387aaa69ba3ec01d1d25ea3dd5b1d","Type":"ContainerStarted","Data":"89d1872498e52c623250c0e179f7f305e7f38fc3609c96c3ef7041ca703326f4"}
Sep 10 15:42:11.234929 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:11.234660    2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:42:12.366668 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:12.366633    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs\") pod \"network-metrics-daemon-tc4ns\" (UID: \"bde0d086-b64d-4188-b17c-4a69af69c9a5\") " pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:12.367132 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:12.366694    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r5qf\" (UniqueName: \"kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf\") pod \"network-check-target-jj8fc\" (UID: \"719ee58f-5767-4813-9b76-0ab62b2c8f3a\") " pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:12.367132 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:12.366718    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:12.367132 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:12.366792    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs podName:bde0d086-b64d-4188-b17c-4a69af69c9a5 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:16.366772416 +0000 UTC m=+9.039182496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs") pod "network-metrics-daemon-tc4ns" (UID: "bde0d086-b64d-4188-b17c-4a69af69c9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:12.367132 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:12.366867    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:12.367132 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:12.366889    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:12.367132 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:12.366903    2572 projected.go:196] Error preparing data for projected volume kube-api-access-5r5qf for pod openshift-network-diagnostics/network-check-target-jj8fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:12.367132 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:12.366954    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf podName:719ee58f-5767-4813-9b76-0ab62b2c8f3a nodeName:}" failed. No retries permitted until 2026-09-10 15:42:16.366936702 +0000 UTC m=+9.039346780 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5r5qf" (UniqueName: "kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf") pod "network-check-target-jj8fc" (UID: "719ee58f-5767-4813-9b76-0ab62b2c8f3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:12.467798 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:12.467762    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret\") pod \"global-pull-secret-syncer-vz9z7\" (UID: \"a7d7c20f-a31c-4468-a822-a7e2601fe3e4\") " pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:12.467979 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:12.467915    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:12.468036 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:12.467982    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret podName:a7d7c20f-a31c-4468-a822-a7e2601fe3e4 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:16.467960333 +0000 UTC m=+9.140370417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret") pod "global-pull-secret-syncer-vz9z7" (UID: "a7d7c20f-a31c-4468-a822-a7e2601fe3e4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:12.987269 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:12.986475    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:12.987269 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:12.986603    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:12.987269 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:12.986957    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:12.987269 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:12.987054    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:12.987269 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:12.987119    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:12.987269 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:12.987176    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:14.986485 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:14.986444    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:14.986965 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:14.986444    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:14.986965 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:14.986649    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:14.986965 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:14.986692    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:14.986965 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:14.986446    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:14.986965 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:14.986784    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:16.403169 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:16.402813    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs\") pod \"network-metrics-daemon-tc4ns\" (UID: \"bde0d086-b64d-4188-b17c-4a69af69c9a5\") " pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:16.403169 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:16.402865    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r5qf\" (UniqueName: \"kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf\") pod \"network-check-target-jj8fc\" (UID: \"719ee58f-5767-4813-9b76-0ab62b2c8f3a\") " pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:16.403169 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:16.403069    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:16.403169 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:16.403076    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:16.403169 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:16.403093    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:16.403169 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:16.403106    2572 projected.go:196] Error preparing data for projected volume kube-api-access-5r5qf for pod openshift-network-diagnostics/network-check-target-jj8fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:16.403169 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:16.403157    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs podName:bde0d086-b64d-4188-b17c-4a69af69c9a5 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:24.403137592 +0000 UTC m=+17.075547666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs") pod "network-metrics-daemon-tc4ns" (UID: "bde0d086-b64d-4188-b17c-4a69af69c9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:16.403169 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:16.403173    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf podName:719ee58f-5767-4813-9b76-0ab62b2c8f3a nodeName:}" failed. No retries permitted until 2026-09-10 15:42:24.403165203 +0000 UTC m=+17.075575273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5r5qf" (UniqueName: "kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf") pod "network-check-target-jj8fc" (UID: "719ee58f-5767-4813-9b76-0ab62b2c8f3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:16.503289 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:16.503249    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret\") pod \"global-pull-secret-syncer-vz9z7\" (UID: \"a7d7c20f-a31c-4468-a822-a7e2601fe3e4\") " pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:16.503476 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:16.503460    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:16.503550 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:16.503539    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret podName:a7d7c20f-a31c-4468-a822-a7e2601fe3e4 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:24.503517769 +0000 UTC m=+17.175927851 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret") pod "global-pull-secret-syncer-vz9z7" (UID: "a7d7c20f-a31c-4468-a822-a7e2601fe3e4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:16.986548 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:16.986510    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:16.986706 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:16.986555    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:16.986706 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:16.986555    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:16.986706 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:16.986640    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:16.986841 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:16.986748    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:16.986885 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:16.986855    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:18.986973 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:18.986886    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:18.986973 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:18.986933    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:18.987501 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:18.987000    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:18.987501 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:18.987013    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:18.987501 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:18.987094    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:18.987501 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:18.987199    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:20.986846 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:20.986804    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:20.986846 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:20.986828    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:20.987395 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:20.986836    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:20.987395 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:20.986947    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:20.987395 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:20.987008    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:20.987395 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:20.987073    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:22.986227 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:22.986185    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:22.986699 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:22.986185    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:22.986699 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:22.986329    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:22.986699 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:22.986326    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:22.986699 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:22.986430    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:22.986699 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:22.986535    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:24.464789 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:24.464451    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs\") pod \"network-metrics-daemon-tc4ns\" (UID: \"bde0d086-b64d-4188-b17c-4a69af69c9a5\") " pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:24.464789 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:24.464514    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r5qf\" (UniqueName: \"kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf\") pod \"network-check-target-jj8fc\" (UID: \"719ee58f-5767-4813-9b76-0ab62b2c8f3a\") " pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:24.464789 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:24.464602    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:24.464789 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:24.464661    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:24.464789 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:24.464682    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:24.464789 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:24.464691    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs podName:bde0d086-b64d-4188-b17c-4a69af69c9a5 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:40.464668389 +0000 UTC m=+33.137078466 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs") pod "network-metrics-daemon-tc4ns" (UID: "bde0d086-b64d-4188-b17c-4a69af69c9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:24.464789 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:24.464696    2572 projected.go:196] Error preparing data for projected volume kube-api-access-5r5qf for pod openshift-network-diagnostics/network-check-target-jj8fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:24.464789 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:24.464757    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf podName:719ee58f-5767-4813-9b76-0ab62b2c8f3a nodeName:}" failed. No retries permitted until 2026-09-10 15:42:40.464741896 +0000 UTC m=+33.137151960 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5r5qf" (UniqueName: "kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf") pod "network-check-target-jj8fc" (UID: "719ee58f-5767-4813-9b76-0ab62b2c8f3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:24.565253 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:24.565205    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret\") pod \"global-pull-secret-syncer-vz9z7\" (UID: \"a7d7c20f-a31c-4468-a822-a7e2601fe3e4\") " pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:24.565463 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:24.565386    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:24.565534 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:24.565468    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret podName:a7d7c20f-a31c-4468-a822-a7e2601fe3e4 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:40.565448311 +0000 UTC m=+33.237858386 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret") pod "global-pull-secret-syncer-vz9z7" (UID: "a7d7c20f-a31c-4468-a822-a7e2601fe3e4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:24.986763 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:24.986732    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:24.986937 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:24.986733    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:24.986937 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:24.986859    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:24.987070 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:24.986935    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:24.987070 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:24.986732    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:24.987070 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:24.987037    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:26.986887 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:26.986849    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:26.986887 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:26.986880    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:26.987340 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:26.986886    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:26.987340 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:26.986953    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:26.987340 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:26.987060    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:26.987340 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:26.987166    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:28.102700 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:28.102465    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-plx44" event={"ID":"8226797c-2abe-4e64-9da6-580fd99c29d7","Type":"ContainerStarted","Data":"09ab7acaf97061982eb580d262980d5c9e4379dbdad267fd799aff594f6142f1"}
Sep 10 15:42:28.104326 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:28.104273    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7sm72" event={"ID":"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9","Type":"ContainerStarted","Data":"ba3b9a3e83ddb9177404eea3595ef1d78dc6d02177af9290cb8564263be45afd"}
Sep 10 15:42:28.105657 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:28.105637    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-36.ec2.internal" event={"ID":"872387aaa69ba3ec01d1d25ea3dd5b1d","Type":"ContainerStarted","Data":"ac335dcec986e2c3b154fbbff7b250a02c4c38446b5ea67621498c64071407b5"}
Sep 10 15:42:28.121848 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:28.121798    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-plx44" podStartSLOduration=2.429468197 podStartE2EDuration="21.121782857s" totalImagesPullingTime="18.69231466s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:28.12148612 +0000 UTC m=+20.793896207" watchObservedRunningTime="2026-09-10 15:42:28.121782857 +0000 UTC m=+20.794192964"
Sep 10 15:42:28.134324 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:28.134248    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-36.ec2.internal" podStartSLOduration=20.134225929 podStartE2EDuration="20.134225929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:28.133737709 +0000 UTC m=+20.806147793" watchObservedRunningTime="2026-09-10 15:42:28.134225929 +0000 UTC m=+20.806636013"
Sep 10 15:42:28.986591 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:28.986359    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:28.986786 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:28.986359    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:28.986786 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:28.986639    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:28.986786 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:28.986716    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:28.986786 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:28.986367    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:28.986957 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:28.986828    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:29.109878 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.109843    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-54n4f" event={"ID":"1611fac1-d845-4d08-bdbf-72957655ab60","Type":"ContainerStarted","Data":"0b6add297ef13cfe269606097688ee02f25e7d796bf8b6b33bdb6df374ca8a61"}
Sep 10 15:42:29.111831 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.111797    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7zlxb" event={"ID":"ca15df82-de7b-4bf8-a736-01c5537e53d2","Type":"ContainerStarted","Data":"4883492dbd5f6c352b5c179ca1b26d4185377a2c37983f85e9018fdd19f486c8"}
Sep 10 15:42:29.113645 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.113607    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q" event={"ID":"9701fbb2-7db9-40d3-af55-f611b396f4cd","Type":"ContainerStarted","Data":"65949d3a6d156662da9f0cbed3c39ce9fd8b7fe0b7fb1c40806f3af92644b7f1"}
Sep 10 15:42:29.115362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.115321    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nf5db" event={"ID":"7a6d6658-59b9-48e2-83e1-8a2b8750ecf7","Type":"ContainerStarted","Data":"6b6b0dbc666ac55a70bde57abce604f5c476e7a066b4f85126e22c15082298e6"}
Sep 10 15:42:29.117094 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.117069    2572 generic.go:356] "Generic (PLEG): container finished" podID="eacf5dc6-da67-4d9e-b76c-8a2e946bf637" containerID="7b6e6a0d7fb1a7c329593eb8c3ad2771613a356d3a6f612fdcefe86a7370eb29" exitCode=0
Sep 10 15:42:29.117187 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.117161    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zg8x" event={"ID":"eacf5dc6-da67-4d9e-b76c-8a2e946bf637","Type":"ContainerDied","Data":"7b6e6a0d7fb1a7c329593eb8c3ad2771613a356d3a6f612fdcefe86a7370eb29"}
Sep 10 15:42:29.120844 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.120824    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7sm72_8ddd4dc1-1122-41e0-a54a-67fb8008d7b9/ovn-acl-logging/0.log"
Sep 10 15:42:29.121258 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.121232    2572 generic.go:356] "Generic (PLEG): container finished" podID="8ddd4dc1-1122-41e0-a54a-67fb8008d7b9" containerID="4a7fe55d0fb5e25642bf90c21d4509d63e868497813c8576fadbcb968c86a90f" exitCode=1
Sep 10 15:42:29.121357 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.121256    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7sm72" event={"ID":"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9","Type":"ContainerDied","Data":"4a7fe55d0fb5e25642bf90c21d4509d63e868497813c8576fadbcb968c86a90f"}
Sep 10 15:42:29.121357 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.121353    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7sm72" event={"ID":"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9","Type":"ContainerStarted","Data":"2e4785800af00baab6bb940319fc2e7d16c1fd6fd1e8cec5be06327bb4b70307"}
Sep 10 15:42:29.121449 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.121401    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7sm72" event={"ID":"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9","Type":"ContainerStarted","Data":"31d76477e601573ffec3e92b045c4eafa3b3eb5bc4c78077f466f9fec9e24861"}
Sep 10 15:42:29.121449 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.121442    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7sm72" event={"ID":"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9","Type":"ContainerStarted","Data":"e657f64e9d929c3194dc051f00da35b5e51a7cc8ea382d08ec5bfe880c88ce27"}
Sep 10 15:42:29.121548 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.121495    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7sm72" event={"ID":"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9","Type":"ContainerStarted","Data":"ff9cbeea782ed55db64db6cc342e17ec4993cdc7ed0dfbc91348e6b697ca5dbf"}
Sep 10 15:42:29.123084 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.123061    2572 generic.go:356] "Generic (PLEG): container finished" podID="f87d7a97be63e6b3b27b586fea74839e" containerID="f91f9ec4aad5370d7e0e3ccf4fc45733c470d9b5464d659c6b9c7abcf9a07f4b" exitCode=0
Sep 10 15:42:29.123209 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.123184    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal" event={"ID":"f87d7a97be63e6b3b27b586fea74839e","Type":"ContainerDied","Data":"f91f9ec4aad5370d7e0e3ccf4fc45733c470d9b5464d659c6b9c7abcf9a07f4b"}
Sep 10 15:42:29.124934 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.124907    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q944h" event={"ID":"f221b59a-f955-4ce1-a9b4-e4efe942a24a","Type":"ContainerStarted","Data":"bc2b12e5042955805a68b05ff849059d1d8b7ab6f17009bbd13d60ed337a3065"}
Sep 10 15:42:29.127891 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.127814    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-54n4f" podStartSLOduration=8.2701415 podStartE2EDuration="22.127796768s" totalImagesPullingTime="13.857655268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:29.126377632 +0000 UTC m=+21.798787717" watchObservedRunningTime="2026-09-10 15:42:29.127796768 +0000 UTC m=+21.800206851"
Sep 10 15:42:29.142936 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.142870    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nf5db" podStartSLOduration=4.287118216 podStartE2EDuration="22.142848259s" totalImagesPullingTime="17.855730043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:29.141989112 +0000 UTC m=+21.814399195" watchObservedRunningTime="2026-09-10 15:42:29.142848259 +0000 UTC m=+21.815258343"
Sep 10 15:42:29.156266 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.156218    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q944h" podStartSLOduration=3.512063936 podStartE2EDuration="22.156199439s" totalImagesPullingTime="18.644135503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:29.155821559 +0000 UTC m=+21.828231642" watchObservedRunningTime="2026-09-10 15:42:29.156199439 +0000 UTC m=+21.828609521"
Sep 10 15:42:29.214216 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.214161    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7zlxb" podStartSLOduration=3.072764124 podStartE2EDuration="22.214143966s" totalImagesPullingTime="19.141379842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:29.213820907 +0000 UTC m=+21.886230989" watchObservedRunningTime="2026-09-10 15:42:29.214143966 +0000 UTC m=+21.886554048"
Sep 10 15:42:29.383323 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.383283    2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 15:42:29.781464 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.781347    2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T15:42:29.383317254Z","UUID":"a8256e3c-bae2-46c5-8ad1-cfd255a751da","Handler":null,"Name":"","Endpoint":""}
Sep 10 15:42:29.784225 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.784190    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 15:42:29.784225 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:29.784225    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 15:42:30.130814 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:30.130424    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal" event={"ID":"f87d7a97be63e6b3b27b586fea74839e","Type":"ContainerStarted","Data":"2ce206ae7dffd41ab1d5d25145a0b639b6004fbace0075997c1c99550ee2f46b"}
Sep 10 15:42:30.133341 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:30.132858    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t67th" event={"ID":"8a7033d6-0cdc-4ef5-8f25-9fc1b4b7aded","Type":"ContainerStarted","Data":"d26ea7a20300bdc9689e3ebeb43315c3f74299924bee1f127458ddb639b7d11f"}
Sep 10 15:42:30.136890 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:30.136856    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q" event={"ID":"9701fbb2-7db9-40d3-af55-f611b396f4cd","Type":"ContainerStarted","Data":"99a6c214cfda65fcf3e583c00d6b6eb2b09cb3b3b411189ddc56d9073b0c369d"}
Sep 10 15:42:30.146246 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:30.146190    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-36.ec2.internal" podStartSLOduration=22.146169322 podStartE2EDuration="22.146169322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:30.146121114 +0000 UTC m=+22.818531200" watchObservedRunningTime="2026-09-10 15:42:30.146169322 +0000 UTC m=+22.818579404"
Sep 10 15:42:30.161323 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:30.161247    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-t67th" podStartSLOduration=3.56286799 podStartE2EDuration="22.161223096s" totalImagesPullingTime="18.598355106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:30.161131132 +0000 UTC m=+22.833541216" watchObservedRunningTime="2026-09-10 15:42:30.161223096 +0000 UTC m=+22.833633180"
Sep 10 15:42:30.986359 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:30.986253    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:30.986359 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:30.986319    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:30.986359 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:30.986329    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:30.986621 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:30.986403    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:30.986621 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:30.986461    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:30.986621 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:30.986558    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:31.142103 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:31.142056    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q" event={"ID":"9701fbb2-7db9-40d3-af55-f611b396f4cd","Type":"ContainerStarted","Data":"1deab47b562197b56e6a3fc361e3c1f2072cca6fe081dec5763242828a5b9718"}
Sep 10 15:42:31.145804 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:31.145768    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7sm72_8ddd4dc1-1122-41e0-a54a-67fb8008d7b9/ovn-acl-logging/0.log"
Sep 10 15:42:31.146240 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:31.146208    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7sm72" event={"ID":"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9","Type":"ContainerStarted","Data":"8a1d28f444719c41c5806deffbf3fe3d9c09d9c12a542701b30642c8f806ca98"}
Sep 10 15:42:31.158989 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:31.158938    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb59q" podStartSLOduration=4.593538323 podStartE2EDuration="24.158917474s" totalImagesPullingTime="19.565379151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:31.158918461 +0000 UTC m=+23.831328546" watchObservedRunningTime="2026-09-10 15:42:31.158917474 +0000 UTC m=+23.831327534"
Sep 10 15:42:31.827338 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:31.827279    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nf5db"
Sep 10 15:42:31.827961 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:31.827940    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nf5db"
Sep 10 15:42:32.986866 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:32.986841    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:32.987296 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:32.986970    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:32.987296 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:32.986997    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:32.987296 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:32.986997    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:32.987296 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:32.987064    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:32.987296 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:32.987120    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:33.124862 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:33.124240    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-54n4f_1611fac1-d845-4d08-bdbf-72957655ab60/dns-node-resolver/0.log"
Sep 10 15:42:33.152817 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:33.152783    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zg8x" event={"ID":"eacf5dc6-da67-4d9e-b76c-8a2e946bf637","Type":"ContainerStarted","Data":"da889e97a6490f98e95b8a7ac8cfed8edf9a94b6b99901c970006d3aaded3f55"}
Sep 10 15:42:33.156687 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:33.156663    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7sm72_8ddd4dc1-1122-41e0-a54a-67fb8008d7b9/ovn-acl-logging/0.log"
Sep 10 15:42:33.157144 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:33.157114    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7sm72" event={"ID":"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9","Type":"ContainerStarted","Data":"4e33af694c73870db37b91a89973fd6107eb96656119c916b33a676041aeb359"}
Sep 10 15:42:33.157637 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:33.157502    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:33.157637 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:33.157579    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:33.157821 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:33.157764    2572 scope.go:117] "RemoveContainer" containerID="4a7fe55d0fb5e25642bf90c21d4509d63e868497813c8576fadbcb968c86a90f"
Sep 10 15:42:33.174218 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:33.173822    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:33.902371 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:33.902342    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q944h_f221b59a-f955-4ce1-a9b4-e4efe942a24a/node-ca/0.log"
Sep 10 15:42:34.160718 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:34.160622    2572 generic.go:356] "Generic (PLEG): container finished" podID="eacf5dc6-da67-4d9e-b76c-8a2e946bf637" containerID="da889e97a6490f98e95b8a7ac8cfed8edf9a94b6b99901c970006d3aaded3f55" exitCode=0
Sep 10 15:42:34.161146 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:34.160714    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zg8x" event={"ID":"eacf5dc6-da67-4d9e-b76c-8a2e946bf637","Type":"ContainerDied","Data":"da889e97a6490f98e95b8a7ac8cfed8edf9a94b6b99901c970006d3aaded3f55"}
Sep 10 15:42:34.164719 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:34.164703    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7sm72_8ddd4dc1-1122-41e0-a54a-67fb8008d7b9/ovn-acl-logging/0.log"
Sep 10 15:42:34.165096 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:34.165067    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7sm72" event={"ID":"8ddd4dc1-1122-41e0-a54a-67fb8008d7b9","Type":"ContainerStarted","Data":"0cedba8ed0fccd3447720a85a8a7265bb24353743a0493eeb094c8975377ee13"}
Sep 10 15:42:34.165371 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:34.165354    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:34.182005 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:34.181977    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:42:34.211505 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:34.211449    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7sm72" podStartSLOduration=8.459451501 podStartE2EDuration="27.21143085s" totalImagesPullingTime="18.751979349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:34.211008945 +0000 UTC m=+26.883419028" watchObservedRunningTime="2026-09-10 15:42:34.21143085 +0000 UTC m=+26.883840932"
Sep 10 15:42:34.969633 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:34.969589    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vz9z7"]
Sep 10 15:42:34.969791 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:34.969741    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:34.969862 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:34.969823    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:34.972660 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:34.972625    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jj8fc"]
Sep 10 15:42:34.972795 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:34.972735    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:34.972856 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:34.972820    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:34.973508 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:34.973481    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tc4ns"]
Sep 10 15:42:34.973634 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:34.973596    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:34.973702 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:34.973682    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:36.171902 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:36.171684    2572 generic.go:356] "Generic (PLEG): container finished" podID="eacf5dc6-da67-4d9e-b76c-8a2e946bf637" containerID="6aec0477d784fa7a1429d731e248f4913807ce70a5154c8825a4c56ef3acd11a" exitCode=0
Sep 10 15:42:36.172474 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:36.171744    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zg8x" event={"ID":"eacf5dc6-da67-4d9e-b76c-8a2e946bf637","Type":"ContainerDied","Data":"6aec0477d784fa7a1429d731e248f4913807ce70a5154c8825a4c56ef3acd11a"}
Sep 10 15:42:36.767276 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:36.767243    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nf5db"
Sep 10 15:42:36.767532 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:36.767515    2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 15:42:36.767847 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:36.767830    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nf5db"
Sep 10 15:42:36.986321 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:36.986262    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:36.986467 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:36.986332    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:36.986467 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:36.986332    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:36.986467 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:36.986399    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:36.986602 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:36.986504    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:36.986602 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:36.986570    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:37.176889 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:37.176854    2572 generic.go:356] "Generic (PLEG): container finished" podID="eacf5dc6-da67-4d9e-b76c-8a2e946bf637" containerID="e395afb682699b4905f90d2f8908a4655b7d9d700925e267abab030a3447519b" exitCode=0
Sep 10 15:42:37.177254 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:37.176940    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zg8x" event={"ID":"eacf5dc6-da67-4d9e-b76c-8a2e946bf637","Type":"ContainerDied","Data":"e395afb682699b4905f90d2f8908a4655b7d9d700925e267abab030a3447519b"}
Sep 10 15:42:38.986211 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:38.986168    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:38.986211 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:38.986206    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:38.986745 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:38.986294    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:38.986745 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:38.986328    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:38.986745 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:38.986410    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:38.986745 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:38.986469    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:40.482671 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:40.482632    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs\") pod \"network-metrics-daemon-tc4ns\" (UID: \"bde0d086-b64d-4188-b17c-4a69af69c9a5\") " pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:40.483178 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:40.482690    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r5qf\" (UniqueName: \"kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf\") pod \"network-check-target-jj8fc\" (UID: \"719ee58f-5767-4813-9b76-0ab62b2c8f3a\") " pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:40.483178 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:40.482785    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:40.483178 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:40.482808    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:40.483178 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:40.482823    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:40.483178 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:40.482832    2572 projected.go:196] Error preparing data for projected volume kube-api-access-5r5qf for pod openshift-network-diagnostics/network-check-target-jj8fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:40.483178 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:40.482861    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs podName:bde0d086-b64d-4188-b17c-4a69af69c9a5 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:12.482841992 +0000 UTC m=+65.155252070 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs") pod "network-metrics-daemon-tc4ns" (UID: "bde0d086-b64d-4188-b17c-4a69af69c9a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:40.483178 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:40.482882    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf podName:719ee58f-5767-4813-9b76-0ab62b2c8f3a nodeName:}" failed. No retries permitted until 2026-09-10 15:43:12.482872116 +0000 UTC m=+65.155282176 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5r5qf" (UniqueName: "kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf") pod "network-check-target-jj8fc" (UID: "719ee58f-5767-4813-9b76-0ab62b2c8f3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:40.584060 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:40.584028    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret\") pod \"global-pull-secret-syncer-vz9z7\" (UID: \"a7d7c20f-a31c-4468-a822-a7e2601fe3e4\") " pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:40.584229 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:40.584149    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:40.584229 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:40.584204    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret podName:a7d7c20f-a31c-4468-a822-a7e2601fe3e4 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:12.584190715 +0000 UTC m=+65.256600775 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret") pod "global-pull-secret-syncer-vz9z7" (UID: "a7d7c20f-a31c-4468-a822-a7e2601fe3e4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:40.986800 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:40.986772    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:40.986999 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:40.986772    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:40.986999 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:40.986894    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:40.986999 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:40.986780    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:40.986999 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:40.986986    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:40.987191 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:40.987033    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:42.190936 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:42.190903    2572 generic.go:356] "Generic (PLEG): container finished" podID="eacf5dc6-da67-4d9e-b76c-8a2e946bf637" containerID="27f28e63cda50c374c660517c9c809df4cca9e3bf8f6d9680240944d502ef1ed" exitCode=0
Sep 10 15:42:42.191386 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:42.190952    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zg8x" event={"ID":"eacf5dc6-da67-4d9e-b76c-8a2e946bf637","Type":"ContainerDied","Data":"27f28e63cda50c374c660517c9c809df4cca9e3bf8f6d9680240944d502ef1ed"}
Sep 10 15:42:42.986215 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:42.986179    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:42.986215 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:42.986215    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:42.986511 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:42.986181    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:42.986511 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:42.986343    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:42.986511 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:42.986413    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:42.986511 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:42.986480    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:43.195857 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:43.195826    2572 generic.go:356] "Generic (PLEG): container finished" podID="eacf5dc6-da67-4d9e-b76c-8a2e946bf637" containerID="b7df650c69a1d3e616f536d5a69c490709f4ecc31fe609c1da6177ce9b272d90" exitCode=0
Sep 10 15:42:43.196215 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:43.195891    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zg8x" event={"ID":"eacf5dc6-da67-4d9e-b76c-8a2e946bf637","Type":"ContainerDied","Data":"b7df650c69a1d3e616f536d5a69c490709f4ecc31fe609c1da6177ce9b272d90"}
Sep 10 15:42:44.202025 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:44.201840    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zg8x" event={"ID":"eacf5dc6-da67-4d9e-b76c-8a2e946bf637","Type":"ContainerStarted","Data":"45e46495ff79d1c16fa00a5c2ce2453522d3453f3a13b8caaa932ab28a3121ba"}
Sep 10 15:42:44.222485 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:44.222440    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6zg8x" podStartSLOduration=12.716966059 podStartE2EDuration="37.222423324s" totalImagesPullingTime="24.505457265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:44.222293278 +0000 UTC m=+36.894703343" watchObservedRunningTime="2026-09-10 15:42:44.222423324 +0000 UTC m=+36.894833407"
Sep 10 15:42:44.985914 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:44.985881    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:44.986080 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:44.985973    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:44.986080 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:44.985981    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:44.986080 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:44.986069    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:44.986204 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:44.986108    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:44.986238 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:44.986201    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:46.986058 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:46.986013    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:46.986058 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:46.986048    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:46.986598 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:46.986013    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:46.986598 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:46.986147    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:46.986598 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:46.986268    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:46.986598 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:46.986371    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:48.985929 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:48.985892    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:48.985929 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:48.985927    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:48.986451 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:48.986004    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:48.986451 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:48.986004    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:48.986451 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:48.986063    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:48.986451 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:48.986149    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:50.986749 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:50.986709    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:50.987153 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:50.986808    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:50.987153 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:50.986817    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:50.987153 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:50.986837    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:50.987153 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:50.986924    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:50.987153 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:50.986980    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:52.986913 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:52.986881    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:52.986913 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:52.986909    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:52.987445 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:52.986881    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:52.987445 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:52.987017    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:52.987445 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:52.987123    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:52.987445 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:52.987184    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:54.986503 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:54.986467    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:54.986895 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:54.986570    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:54.986895 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:54.986584    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vz9z7" podUID="a7d7c20f-a31c-4468-a822-a7e2601fe3e4"
Sep 10 15:42:54.986895 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:54.986594    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:54.986895 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:54.986649    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jj8fc" podUID="719ee58f-5767-4813-9b76-0ab62b2c8f3a"
Sep 10 15:42:54.986895 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:54.986766    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc4ns" podUID="bde0d086-b64d-4188-b17c-4a69af69c9a5"
Sep 10 15:42:56.094086 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.094058    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-36.ec2.internal" event="NodeReady"
Sep 10 15:42:56.094494 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.094186    2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 15:42:56.129365 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.129334    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d854b8bbc-pv7h9"]
Sep 10 15:42:56.149742 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.149713    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d854b8bbc-pv7h9"]
Sep 10 15:42:56.149888 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.149751    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xrtrf"]
Sep 10 15:42:56.149888 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.149826    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.152646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.152625    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 15:42:56.152764 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.152650    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 15:42:56.153948 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.153929    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 15:42:56.154650 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.154633    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hck2z\""
Sep 10 15:42:56.158256 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.158239    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 15:42:56.173840 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.173812    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nkbvh"]
Sep 10 15:42:56.174108 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.174073    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xrtrf"
Sep 10 15:42:56.176633 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.176611    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-brxch\""
Sep 10 15:42:56.176735 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.176688    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 15:42:56.176735 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.176686    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 15:42:56.195936 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.195915    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xrtrf"]
Sep 10 15:42:56.196030 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.195948    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nkbvh"]
Sep 10 15:42:56.196030 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.195967    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vclm6"]
Sep 10 15:42:56.196030 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.196015    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nkbvh"
Sep 10 15:42:56.198058 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.198035    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 15:42:56.198209 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.198193    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pdj5g\""
Sep 10 15:42:56.198209 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.198203    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 15:42:56.198449 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.198436    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 15:42:56.220013 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.219988    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vclm6"]
Sep 10 15:42:56.220108 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.220093    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vclm6"
Sep 10 15:42:56.222511 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.222485    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 15:42:56.222511 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.222500    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 15:42:56.222680 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.222500    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 15:42:56.222680 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.222551    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5dvnv\""
Sep 10 15:42:56.222680 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.222583    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 15:42:56.291564 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.291530    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97b545a8-3fac-4bcf-b67c-f51a919e4508-config-volume\") pod \"dns-default-xrtrf\" (UID: \"97b545a8-3fac-4bcf-b67c-f51a919e4508\") " pod="openshift-dns/dns-default-xrtrf"
Sep 10 15:42:56.291564 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.291566    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6ds9\" (UniqueName: \"kubernetes.io/projected/6b4a2209-e010-48a8-b869-89467731e086-kube-api-access-g6ds9\") pod \"ingress-canary-nkbvh\" (UID: \"6b4a2209-e010-48a8-b869-89467731e086\") " pod="openshift-ingress-canary/ingress-canary-nkbvh"
Sep 10 15:42:56.291748 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.291597    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxp92\" (UniqueName: \"kubernetes.io/projected/721c0590-d431-4abf-b9b6-fd5ab568ddcd-kube-api-access-dxp92\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.291748 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.291664    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b4a2209-e010-48a8-b869-89467731e086-cert\") pod \"ingress-canary-nkbvh\" (UID: \"6b4a2209-e010-48a8-b869-89467731e086\") " pod="openshift-ingress-canary/ingress-canary-nkbvh"
Sep 10 15:42:56.291748 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.291701    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/721c0590-d431-4abf-b9b6-fd5ab568ddcd-installation-pull-secrets\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.291748 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.291723    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/721c0590-d431-4abf-b9b6-fd5ab568ddcd-bound-sa-token\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.291748 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.291742    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/721c0590-d431-4abf-b9b6-fd5ab568ddcd-ca-trust-extracted\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.291897 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.291761    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/721c0590-d431-4abf-b9b6-fd5ab568ddcd-trusted-ca\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.291897 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.291783    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97b545a8-3fac-4bcf-b67c-f51a919e4508-metrics-tls\") pod \"dns-default-xrtrf\" (UID: \"97b545a8-3fac-4bcf-b67c-f51a919e4508\") " pod="openshift-dns/dns-default-xrtrf"
Sep 10 15:42:56.291897 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.291796    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b545a8-3fac-4bcf-b67c-f51a919e4508-tmp-dir\") pod \"dns-default-xrtrf\" (UID: \"97b545a8-3fac-4bcf-b67c-f51a919e4508\") " pod="openshift-dns/dns-default-xrtrf"
Sep 10 15:42:56.291897 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.291827    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/721c0590-d431-4abf-b9b6-fd5ab568ddcd-image-registry-private-configuration\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.291897 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.291851    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/721c0590-d431-4abf-b9b6-fd5ab568ddcd-registry-tls\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.291897 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.291867    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/721c0590-d431-4abf-b9b6-fd5ab568ddcd-registry-certificates\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.291897 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.291883    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2czv\" (UniqueName: \"kubernetes.io/projected/97b545a8-3fac-4bcf-b67c-f51a919e4508-kube-api-access-g2czv\") pod \"dns-default-xrtrf\" (UID: \"97b545a8-3fac-4bcf-b67c-f51a919e4508\") " pod="openshift-dns/dns-default-xrtrf"
Sep 10 15:42:56.392418 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392319    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97b545a8-3fac-4bcf-b67c-f51a919e4508-config-volume\") pod \"dns-default-xrtrf\" (UID: \"97b545a8-3fac-4bcf-b67c-f51a919e4508\") " pod="openshift-dns/dns-default-xrtrf"
Sep 10 15:42:56.392418 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392352    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6ds9\" (UniqueName: \"kubernetes.io/projected/6b4a2209-e010-48a8-b869-89467731e086-kube-api-access-g6ds9\") pod \"ingress-canary-nkbvh\" (UID: \"6b4a2209-e010-48a8-b869-89467731e086\") " pod="openshift-ingress-canary/ingress-canary-nkbvh"
Sep 10 15:42:56.392418 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392375    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/06cdbeec-2930-46fd-bf24-51c86356c801-data-volume\") pod \"insights-runtime-extractor-vclm6\" (UID: \"06cdbeec-2930-46fd-bf24-51c86356c801\") " pod="openshift-insights/insights-runtime-extractor-vclm6"
Sep 10 15:42:56.392418 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392405    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxp92\" (UniqueName: \"kubernetes.io/projected/721c0590-d431-4abf-b9b6-fd5ab568ddcd-kube-api-access-dxp92\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.392743 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392429    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/06cdbeec-2930-46fd-bf24-51c86356c801-crio-socket\") pod \"insights-runtime-extractor-vclm6\" (UID: \"06cdbeec-2930-46fd-bf24-51c86356c801\") " pod="openshift-insights/insights-runtime-extractor-vclm6"
Sep 10 15:42:56.392743 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392542    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b4a2209-e010-48a8-b869-89467731e086-cert\") pod \"ingress-canary-nkbvh\" (UID: \"6b4a2209-e010-48a8-b869-89467731e086\") " pod="openshift-ingress-canary/ingress-canary-nkbvh"
Sep 10 15:42:56.392743 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392572    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/721c0590-d431-4abf-b9b6-fd5ab568ddcd-installation-pull-secrets\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.392743 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392597    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkqsl\" (UniqueName: \"kubernetes.io/projected/06cdbeec-2930-46fd-bf24-51c86356c801-kube-api-access-bkqsl\") pod \"insights-runtime-extractor-vclm6\" (UID: \"06cdbeec-2930-46fd-bf24-51c86356c801\") " pod="openshift-insights/insights-runtime-extractor-vclm6"
Sep 10 15:42:56.392743 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392679    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/721c0590-d431-4abf-b9b6-fd5ab568ddcd-bound-sa-token\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.392743 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392724    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/721c0590-d431-4abf-b9b6-fd5ab568ddcd-ca-trust-extracted\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.393011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392752    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/721c0590-d431-4abf-b9b6-fd5ab568ddcd-trusted-ca\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.393011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392780    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97b545a8-3fac-4bcf-b67c-f51a919e4508-metrics-tls\") pod \"dns-default-xrtrf\" (UID: \"97b545a8-3fac-4bcf-b67c-f51a919e4508\") " pod="openshift-dns/dns-default-xrtrf"
Sep 10 15:42:56.393011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392805    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b545a8-3fac-4bcf-b67c-f51a919e4508-tmp-dir\") pod \"dns-default-xrtrf\" (UID: \"97b545a8-3fac-4bcf-b67c-f51a919e4508\") " pod="openshift-dns/dns-default-xrtrf"
Sep 10 15:42:56.393011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392836    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06cdbeec-2930-46fd-bf24-51c86356c801-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vclm6\" (UID: \"06cdbeec-2930-46fd-bf24-51c86356c801\") " pod="openshift-insights/insights-runtime-extractor-vclm6"
Sep 10 15:42:56.393011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392867    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06cdbeec-2930-46fd-bf24-51c86356c801-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vclm6\" (UID: \"06cdbeec-2930-46fd-bf24-51c86356c801\") " pod="openshift-insights/insights-runtime-extractor-vclm6"
Sep 10 15:42:56.393011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392909    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/721c0590-d431-4abf-b9b6-fd5ab568ddcd-image-registry-private-configuration\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.393011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392936    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/721c0590-d431-4abf-b9b6-fd5ab568ddcd-registry-tls\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.393011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392959    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/721c0590-d431-4abf-b9b6-fd5ab568ddcd-registry-certificates\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.393011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392976    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97b545a8-3fac-4bcf-b67c-f51a919e4508-config-volume\") pod \"dns-default-xrtrf\" (UID: \"97b545a8-3fac-4bcf-b67c-f51a919e4508\") " pod="openshift-dns/dns-default-xrtrf"
Sep 10 15:42:56.393011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.392988    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2czv\" (UniqueName: \"kubernetes.io/projected/97b545a8-3fac-4bcf-b67c-f51a919e4508-kube-api-access-g2czv\") pod \"dns-default-xrtrf\" (UID: \"97b545a8-3fac-4bcf-b67c-f51a919e4508\") " pod="openshift-dns/dns-default-xrtrf"
Sep 10 15:42:56.393482 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.393120    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/721c0590-d431-4abf-b9b6-fd5ab568ddcd-ca-trust-extracted\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.393482 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.393289    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b545a8-3fac-4bcf-b67c-f51a919e4508-tmp-dir\") pod \"dns-default-xrtrf\" (UID: \"97b545a8-3fac-4bcf-b67c-f51a919e4508\") " pod="openshift-dns/dns-default-xrtrf"
Sep 10 15:42:56.393772 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.393735    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/721c0590-d431-4abf-b9b6-fd5ab568ddcd-trusted-ca\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.393947 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.393927    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/721c0590-d431-4abf-b9b6-fd5ab568ddcd-registry-certificates\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.396770 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.396744    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97b545a8-3fac-4bcf-b67c-f51a919e4508-metrics-tls\") pod \"dns-default-xrtrf\" (UID: \"97b545a8-3fac-4bcf-b67c-f51a919e4508\") " pod="openshift-dns/dns-default-xrtrf"
Sep 10 15:42:56.396865 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.396817    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/721c0590-d431-4abf-b9b6-fd5ab568ddcd-registry-tls\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.396865 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.396855    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/721c0590-d431-4abf-b9b6-fd5ab568ddcd-installation-pull-secrets\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.396982 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.396872    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b4a2209-e010-48a8-b869-89467731e086-cert\") pod \"ingress-canary-nkbvh\" (UID: \"6b4a2209-e010-48a8-b869-89467731e086\") " pod="openshift-ingress-canary/ingress-canary-nkbvh"
Sep 10 15:42:56.396982 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.396901    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/721c0590-d431-4abf-b9b6-fd5ab568ddcd-image-registry-private-configuration\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.400294 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.400265    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6ds9\" (UniqueName: \"kubernetes.io/projected/6b4a2209-e010-48a8-b869-89467731e086-kube-api-access-g6ds9\") pod \"ingress-canary-nkbvh\" (UID: \"6b4a2209-e010-48a8-b869-89467731e086\") " pod="openshift-ingress-canary/ingress-canary-nkbvh"
Sep 10 15:42:56.400390 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.400349    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2czv\" (UniqueName: \"kubernetes.io/projected/97b545a8-3fac-4bcf-b67c-f51a919e4508-kube-api-access-g2czv\") pod \"dns-default-xrtrf\" (UID: \"97b545a8-3fac-4bcf-b67c-f51a919e4508\") " pod="openshift-dns/dns-default-xrtrf"
Sep 10 15:42:56.400873 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.400857    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/721c0590-d431-4abf-b9b6-fd5ab568ddcd-bound-sa-token\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.401483 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.401462    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxp92\" (UniqueName: \"kubernetes.io/projected/721c0590-d431-4abf-b9b6-fd5ab568ddcd-kube-api-access-dxp92\") pod \"image-registry-5d854b8bbc-pv7h9\" (UID: \"721c0590-d431-4abf-b9b6-fd5ab568ddcd\") " pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.459490 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.459454    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:56.483649 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.483618    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xrtrf"
Sep 10 15:42:56.493900 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.493869    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06cdbeec-2930-46fd-bf24-51c86356c801-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vclm6\" (UID: \"06cdbeec-2930-46fd-bf24-51c86356c801\") " pod="openshift-insights/insights-runtime-extractor-vclm6"
Sep 10 15:42:56.494015 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.493904    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06cdbeec-2930-46fd-bf24-51c86356c801-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vclm6\" (UID: \"06cdbeec-2930-46fd-bf24-51c86356c801\") " pod="openshift-insights/insights-runtime-extractor-vclm6"
Sep 10 15:42:56.494015 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.493960    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/06cdbeec-2930-46fd-bf24-51c86356c801-data-volume\") pod \"insights-runtime-extractor-vclm6\" (UID: \"06cdbeec-2930-46fd-bf24-51c86356c801\") " pod="openshift-insights/insights-runtime-extractor-vclm6"
Sep 10 15:42:56.494015 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.493988    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/06cdbeec-2930-46fd-bf24-51c86356c801-crio-socket\") pod \"insights-runtime-extractor-vclm6\" (UID: \"06cdbeec-2930-46fd-bf24-51c86356c801\") " pod="openshift-insights/insights-runtime-extractor-vclm6"
Sep 10 15:42:56.494175 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.494029    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkqsl\" (UniqueName: \"kubernetes.io/projected/06cdbeec-2930-46fd-bf24-51c86356c801-kube-api-access-bkqsl\") pod \"insights-runtime-extractor-vclm6\" (UID: \"06cdbeec-2930-46fd-bf24-51c86356c801\") " pod="openshift-insights/insights-runtime-extractor-vclm6"
Sep 10 15:42:56.494234 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.494207    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/06cdbeec-2930-46fd-bf24-51c86356c801-crio-socket\") pod \"insights-runtime-extractor-vclm6\" (UID: \"06cdbeec-2930-46fd-bf24-51c86356c801\") " pod="openshift-insights/insights-runtime-extractor-vclm6"
Sep 10 15:42:56.494565 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.494436    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/06cdbeec-2930-46fd-bf24-51c86356c801-data-volume\") pod \"insights-runtime-extractor-vclm6\" (UID: \"06cdbeec-2930-46fd-bf24-51c86356c801\") " pod="openshift-insights/insights-runtime-extractor-vclm6"
Sep 10 15:42:56.494565 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.494504    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06cdbeec-2930-46fd-bf24-51c86356c801-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vclm6\" (UID: \"06cdbeec-2930-46fd-bf24-51c86356c801\") " pod="openshift-insights/insights-runtime-extractor-vclm6"
Sep 10 15:42:56.496339 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.496295    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06cdbeec-2930-46fd-bf24-51c86356c801-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vclm6\" (UID: \"06cdbeec-2930-46fd-bf24-51c86356c801\") " pod="openshift-insights/insights-runtime-extractor-vclm6"
Sep 10 15:42:56.504227 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.504203    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nkbvh"
Sep 10 15:42:56.511925 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.511904    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkqsl\" (UniqueName: \"kubernetes.io/projected/06cdbeec-2930-46fd-bf24-51c86356c801-kube-api-access-bkqsl\") pod \"insights-runtime-extractor-vclm6\" (UID: \"06cdbeec-2930-46fd-bf24-51c86356c801\") " pod="openshift-insights/insights-runtime-extractor-vclm6"
Sep 10 15:42:56.527924 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.527898    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vclm6"
Sep 10 15:42:56.784151 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.784078    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xrtrf"]
Sep 10 15:42:56.787628 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:56.787598    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b545a8_3fac_4bcf_b67c_f51a919e4508.slice/crio-da4e56ac94ecec4b192f6b5f1c6bbb13d2befde4a41b3e35bcbc31edb8e266b6 WatchSource:0}: Error finding container da4e56ac94ecec4b192f6b5f1c6bbb13d2befde4a41b3e35bcbc31edb8e266b6: Status 404 returned error can't find the container with id da4e56ac94ecec4b192f6b5f1c6bbb13d2befde4a41b3e35bcbc31edb8e266b6
Sep 10 15:42:56.791738 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.791712    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d854b8bbc-pv7h9"]
Sep 10 15:42:56.795117 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:56.795093    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721c0590_d431_4abf_b9b6_fd5ab568ddcd.slice/crio-a69c1abe4cb4f8120b910257b5c8b5bd951acba3d9d3cbc8f86a1e55cce17c52 WatchSource:0}: Error finding container a69c1abe4cb4f8120b910257b5c8b5bd951acba3d9d3cbc8f86a1e55cce17c52: Status 404 returned error can't find the container with id a69c1abe4cb4f8120b910257b5c8b5bd951acba3d9d3cbc8f86a1e55cce17c52
Sep 10 15:42:56.818605 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.818580    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nkbvh"]
Sep 10 15:42:56.822016 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:56.821981    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b4a2209_e010_48a8_b869_89467731e086.slice/crio-49acbd28d7df753beed7a29259b2a536ba3cf1f3d33d6b854e5c7483b172a83e WatchSource:0}: Error finding container 49acbd28d7df753beed7a29259b2a536ba3cf1f3d33d6b854e5c7483b172a83e: Status 404 returned error can't find the container with id 49acbd28d7df753beed7a29259b2a536ba3cf1f3d33d6b854e5c7483b172a83e
Sep 10 15:42:56.841021 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.840999    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vclm6"]
Sep 10 15:42:56.850214 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:42:56.850191    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06cdbeec_2930_46fd_bf24_51c86356c801.slice/crio-54a032211f7c775c241d0cd6f73a2ab664d1d190992ff523d6ce9bc455f98609 WatchSource:0}: Error finding container 54a032211f7c775c241d0cd6f73a2ab664d1d190992ff523d6ce9bc455f98609: Status 404 returned error can't find the container with id 54a032211f7c775c241d0cd6f73a2ab664d1d190992ff523d6ce9bc455f98609
Sep 10 15:42:56.986718 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.986681    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:42:56.986902 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.986785    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:42:56.986967 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.986924    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:42:56.989503 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.989480    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 15:42:56.989612 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.989480    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 15:42:56.989612 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.989588    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2gdj6\""
Sep 10 15:42:56.989714 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.989676    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sjjtv\""
Sep 10 15:42:56.989879 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.989862    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 15:42:56.989931 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:56.989862    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 15:42:57.234189 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:57.233404    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vclm6" event={"ID":"06cdbeec-2930-46fd-bf24-51c86356c801","Type":"ContainerStarted","Data":"44acf7c3e77e526a8d81ed3a468b22b4b8ff274c087c3099cce71eff7dd3526b"}
Sep 10 15:42:57.234189 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:57.233519    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vclm6" event={"ID":"06cdbeec-2930-46fd-bf24-51c86356c801","Type":"ContainerStarted","Data":"54a032211f7c775c241d0cd6f73a2ab664d1d190992ff523d6ce9bc455f98609"}
Sep 10 15:42:57.235494 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:57.235465    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nkbvh" event={"ID":"6b4a2209-e010-48a8-b869-89467731e086","Type":"ContainerStarted","Data":"49acbd28d7df753beed7a29259b2a536ba3cf1f3d33d6b854e5c7483b172a83e"}
Sep 10 15:42:57.236951 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:57.236921    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9" event={"ID":"721c0590-d431-4abf-b9b6-fd5ab568ddcd","Type":"ContainerStarted","Data":"da0ed35190915e79b79e4706d3515fc35674374ac0ffdc4be01d749f5a7c58e1"}
Sep 10 15:42:57.237063 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:57.236965    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9" event={"ID":"721c0590-d431-4abf-b9b6-fd5ab568ddcd","Type":"ContainerStarted","Data":"a69c1abe4cb4f8120b910257b5c8b5bd951acba3d9d3cbc8f86a1e55cce17c52"}
Sep 10 15:42:57.237063 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:57.237032    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:42:57.238051 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:57.238032    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xrtrf" event={"ID":"97b545a8-3fac-4bcf-b67c-f51a919e4508","Type":"ContainerStarted","Data":"da4e56ac94ecec4b192f6b5f1c6bbb13d2befde4a41b3e35bcbc31edb8e266b6"}
Sep 10 15:42:57.256312 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:57.256257    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9" podStartSLOduration=2.2562416020000002 podStartE2EDuration="2.256241602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:57.255352118 +0000 UTC m=+49.927762206" watchObservedRunningTime="2026-09-10 15:42:57.256241602 +0000 UTC m=+49.928651685"
Sep 10 15:42:58.243876 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:58.243839    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vclm6" event={"ID":"06cdbeec-2930-46fd-bf24-51c86356c801","Type":"ContainerStarted","Data":"b42c18ee2cff71df6f443a715b1856720759cc6a13cf8df07891054eeb16ac7a"}
Sep 10 15:42:59.322683 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.322647    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-xv7bn"]
Sep 10 15:42:59.325122 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.325092    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-xv7bn"
Sep 10 15:42:59.338924 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.338901    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 10 15:42:59.339052 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.339032    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 15:42:59.339826 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.339799    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 15:42:59.339930 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.339918    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 15:42:59.340030 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.340013    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bpmmb\""
Sep 10 15:42:59.340144 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.340123    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 10 15:42:59.357288 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.357264    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-xv7bn"]
Sep 10 15:42:59.416603 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.416564    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fc7de00-4d51-42c5-b8dd-7077840e7d79-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-xv7bn\" (UID: \"6fc7de00-4d51-42c5-b8dd-7077840e7d79\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xv7bn"
Sep 10 15:42:59.416603 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.416607    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g75zl\" (UniqueName: \"kubernetes.io/projected/6fc7de00-4d51-42c5-b8dd-7077840e7d79-kube-api-access-g75zl\") pod \"prometheus-operator-6f4946db56-xv7bn\" (UID: \"6fc7de00-4d51-42c5-b8dd-7077840e7d79\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xv7bn"
Sep 10 15:42:59.416812 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.416762    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fc7de00-4d51-42c5-b8dd-7077840e7d79-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-xv7bn\" (UID: \"6fc7de00-4d51-42c5-b8dd-7077840e7d79\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xv7bn"
Sep 10 15:42:59.416812 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.416800    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fc7de00-4d51-42c5-b8dd-7077840e7d79-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-xv7bn\" (UID: \"6fc7de00-4d51-42c5-b8dd-7077840e7d79\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xv7bn"
Sep 10 15:42:59.518079 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.518030    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fc7de00-4d51-42c5-b8dd-7077840e7d79-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-xv7bn\" (UID: \"6fc7de00-4d51-42c5-b8dd-7077840e7d79\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xv7bn"
Sep 10 15:42:59.518079 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.518077    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g75zl\" (UniqueName: \"kubernetes.io/projected/6fc7de00-4d51-42c5-b8dd-7077840e7d79-kube-api-access-g75zl\") pod \"prometheus-operator-6f4946db56-xv7bn\" (UID: \"6fc7de00-4d51-42c5-b8dd-7077840e7d79\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xv7bn"
Sep 10 15:42:59.518356 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.518327    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fc7de00-4d51-42c5-b8dd-7077840e7d79-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-xv7bn\" (UID: \"6fc7de00-4d51-42c5-b8dd-7077840e7d79\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xv7bn"
Sep 10 15:42:59.518435 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.518362    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fc7de00-4d51-42c5-b8dd-7077840e7d79-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-xv7bn\" (UID: \"6fc7de00-4d51-42c5-b8dd-7077840e7d79\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xv7bn"
Sep 10 15:42:59.518575 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:59.518548    2572 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 10 15:42:59.518734 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:42:59.518633    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fc7de00-4d51-42c5-b8dd-7077840e7d79-prometheus-operator-tls podName:6fc7de00-4d51-42c5-b8dd-7077840e7d79 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:00.018610443 +0000 UTC m=+52.691020506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/6fc7de00-4d51-42c5-b8dd-7077840e7d79-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-xv7bn" (UID: "6fc7de00-4d51-42c5-b8dd-7077840e7d79") : secret "prometheus-operator-tls" not found
Sep 10 15:42:59.519099 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.519058    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fc7de00-4d51-42c5-b8dd-7077840e7d79-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-xv7bn\" (UID: \"6fc7de00-4d51-42c5-b8dd-7077840e7d79\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xv7bn"
Sep 10 15:42:59.521160 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.521135    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fc7de00-4d51-42c5-b8dd-7077840e7d79-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-xv7bn\" (UID: \"6fc7de00-4d51-42c5-b8dd-7077840e7d79\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xv7bn"
Sep 10 15:42:59.526801 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:42:59.526766    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g75zl\" (UniqueName: \"kubernetes.io/projected/6fc7de00-4d51-42c5-b8dd-7077840e7d79-kube-api-access-g75zl\") pod \"prometheus-operator-6f4946db56-xv7bn\" (UID: \"6fc7de00-4d51-42c5-b8dd-7077840e7d79\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xv7bn"
Sep 10 15:43:00.021338 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:00.021219    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fc7de00-4d51-42c5-b8dd-7077840e7d79-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-xv7bn\" (UID: \"6fc7de00-4d51-42c5-b8dd-7077840e7d79\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xv7bn"
Sep 10 15:43:00.023595 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:00.023570    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fc7de00-4d51-42c5-b8dd-7077840e7d79-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-xv7bn\" (UID: \"6fc7de00-4d51-42c5-b8dd-7077840e7d79\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xv7bn"
Sep 10 15:43:00.236932 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:00.236886    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-xv7bn"
Sep 10 15:43:00.253388 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:00.253344    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xrtrf" event={"ID":"97b545a8-3fac-4bcf-b67c-f51a919e4508","Type":"ContainerStarted","Data":"553eb75e40ade0c4e99372a2354196c303124d4f5d2b38f864a35f09d69a8c21"}
Sep 10 15:43:00.253530 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:00.253409    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xrtrf" event={"ID":"97b545a8-3fac-4bcf-b67c-f51a919e4508","Type":"ContainerStarted","Data":"38cf7a32f432275a13d97305c9b5cf10835059105fd5b3ae364c479ceb4c0ca7"}
Sep 10 15:43:00.253530 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:00.253442    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xrtrf"
Sep 10 15:43:00.255647 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:00.255614    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vclm6" event={"ID":"06cdbeec-2930-46fd-bf24-51c86356c801","Type":"ContainerStarted","Data":"29564aaca9ac7b06d8de78a42c36310c525e3a3f6797a69f359ef0c5a6b2d1a8"}
Sep 10 15:43:00.257254 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:00.257227    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nkbvh" event={"ID":"6b4a2209-e010-48a8-b869-89467731e086","Type":"ContainerStarted","Data":"7ad27d4d77ef97fe727fb9fdc92be560e8de923aa23bd3d987e9d13c686a80b5"}
Sep 10 15:43:00.271629 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:00.271535    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xrtrf" podStartSLOduration=1.318066431 podStartE2EDuration="4.271519401s" totalImagesPullingTime="2.95345297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:00.270270429 +0000 UTC m=+52.942680525" watchObservedRunningTime="2026-09-10 15:43:00.271519401 +0000 UTC m=+52.943929532"
Sep 10 15:43:00.286904 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:00.286854    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vclm6" podStartSLOduration=1.556859174 podStartE2EDuration="4.286834544s" totalImagesPullingTime="2.72997537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:00.28646346 +0000 UTC m=+52.958873544" watchObservedRunningTime="2026-09-10 15:43:00.286834544 +0000 UTC m=+52.959244627"
Sep 10 15:43:00.301462 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:00.301402    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nkbvh" podStartSLOduration=1.365244463 podStartE2EDuration="4.30138014s" totalImagesPullingTime="2.936135677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:00.300077796 +0000 UTC m=+52.972487880" watchObservedRunningTime="2026-09-10 15:43:00.30138014 +0000 UTC m=+52.973790221"
Sep 10 15:43:00.517624 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:00.517592    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-xv7bn"]
Sep 10 15:43:00.520764 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:43:00.520733    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc7de00_4d51_42c5_b8dd_7077840e7d79.slice/crio-a7de686c2264ea78ea9b64f0a94803cf1e1cdf58e042a20ebaa27dc00821ceca WatchSource:0}: Error finding container a7de686c2264ea78ea9b64f0a94803cf1e1cdf58e042a20ebaa27dc00821ceca: Status 404 returned error can't find the container with id a7de686c2264ea78ea9b64f0a94803cf1e1cdf58e042a20ebaa27dc00821ceca
Sep 10 15:43:01.261834 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:01.261790    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-xv7bn" event={"ID":"6fc7de00-4d51-42c5-b8dd-7077840e7d79","Type":"ContainerStarted","Data":"a7de686c2264ea78ea9b64f0a94803cf1e1cdf58e042a20ebaa27dc00821ceca"}
Sep 10 15:43:02.269089 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:02.269054    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-xv7bn" event={"ID":"6fc7de00-4d51-42c5-b8dd-7077840e7d79","Type":"ContainerStarted","Data":"10ea634e05b55c9805e99aa242e5938c420dfbbd1bd42b17ea5546069023ed46"}
Sep 10 15:43:02.269482 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:02.269147    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-xv7bn" event={"ID":"6fc7de00-4d51-42c5-b8dd-7077840e7d79","Type":"ContainerStarted","Data":"15e900744e5c33a64dd351b1ed783a765f294d7c6b25d56de2d3c5cc99660bab"}
Sep 10 15:43:02.285049 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:02.284955    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-xv7bn" podStartSLOduration=1.773734003 podStartE2EDuration="3.284941791s" totalImagesPullingTime="1.511207788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:02.283685577 +0000 UTC m=+54.956095671" watchObservedRunningTime="2026-09-10 15:43:02.284941791 +0000 UTC m=+54.957351871"
Sep 10 15:43:04.697518 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.697483    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-96zqr"]
Sep 10 15:43:04.738630 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.738600    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.741540 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.741509    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 15:43:04.741673 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.741565    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 15:43:04.742113 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.742098    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 15:43:04.742164 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.742112    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-j55xj\""
Sep 10 15:43:04.862090 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.862055    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-node-exporter-textfile\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.862090 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.862098    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-node-exporter-accelerators-collector-config\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.862351 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.862133    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.862351 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.862198    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-root\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.862351 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.862237    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-metrics-client-ca\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.862351 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.862257    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-node-exporter-tls\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.862351 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.862280    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lft7\" (UniqueName: \"kubernetes.io/projected/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-kube-api-access-7lft7\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.862351 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.862325    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-sys\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.862620 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.862469    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-node-exporter-wtmp\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.963792 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.963691    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-node-exporter-wtmp\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.963792 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.963743    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-node-exporter-textfile\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.963792 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.963766    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-node-exporter-accelerators-collector-config\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.964056 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.963816    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.964056 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.963850    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-root\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.964056 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.963861    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-node-exporter-wtmp\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.964056 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.963888    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-metrics-client-ca\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.964056 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.963927    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-node-exporter-tls\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.964056 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.963931    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-root\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.964056 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.963978    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lft7\" (UniqueName: \"kubernetes.io/projected/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-kube-api-access-7lft7\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.964056 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.964006    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-sys\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.964490 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.964114    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-sys\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.964490 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.964117    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-node-exporter-textfile\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.964490 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.964429    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-node-exporter-accelerators-collector-config\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.964631 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.964550    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-metrics-client-ca\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.966240 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.966219    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-node-exporter-tls\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.966366 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.966240    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:04.972056 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:04.972028    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lft7\" (UniqueName: \"kubernetes.io/projected/ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f-kube-api-access-7lft7\") pod \"node-exporter-96zqr\" (UID: \"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f\") " pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:05.047870 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:05.047829    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-96zqr"
Sep 10 15:43:05.087242 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:43:05.087211    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6b3fa4_7917_4dd5_bc85_7b6bdbe3802f.slice/crio-c6a628551be9fefebac873e5993727b781c49f9bc365e775a9cfaf52db0ab0da WatchSource:0}: Error finding container c6a628551be9fefebac873e5993727b781c49f9bc365e775a9cfaf52db0ab0da: Status 404 returned error can't find the container with id c6a628551be9fefebac873e5993727b781c49f9bc365e775a9cfaf52db0ab0da
Sep 10 15:43:05.279579 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:05.279489    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-96zqr" event={"ID":"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f","Type":"ContainerStarted","Data":"c6a628551be9fefebac873e5993727b781c49f9bc365e775a9cfaf52db0ab0da"}
Sep 10 15:43:06.183032 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:06.183006    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7sm72"
Sep 10 15:43:06.283940 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:06.283893    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-96zqr" event={"ID":"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f","Type":"ContainerStarted","Data":"894288cd1148f9d3cbda1382d8dc299737770b7c7ec9e512557650aac1e731c0"}
Sep 10 15:43:07.288720 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.288686    2572 generic.go:356] "Generic (PLEG): container finished" podID="ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f" containerID="894288cd1148f9d3cbda1382d8dc299737770b7c7ec9e512557650aac1e731c0" exitCode=0
Sep 10 15:43:07.289103 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.288775    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-96zqr" event={"ID":"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f","Type":"ContainerDied","Data":"894288cd1148f9d3cbda1382d8dc299737770b7c7ec9e512557650aac1e731c0"}
Sep 10 15:43:07.780461 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.780431    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-b74c4f759-qjbvs"]
Sep 10 15:43:07.783725 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.783699    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.787820 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.787793    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 10 15:43:07.787990 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.787968    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-f5qlq\""
Sep 10 15:43:07.788072 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.787986    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ni5p8lhjgf5v\""
Sep 10 15:43:07.788399 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.788380    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 10 15:43:07.788503 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.788449    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 10 15:43:07.788503 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.788481    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 10 15:43:07.788605 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.788505    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 10 15:43:07.803902 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.803878    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b74c4f759-qjbvs"]
Sep 10 15:43:07.889022 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.888983    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/040f0003-1267-49e6-a365-70608c4bc7d1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.889022 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.889026    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/040f0003-1267-49e6-a365-70608c4bc7d1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.889221 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.889050    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/040f0003-1267-49e6-a365-70608c4bc7d1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.889221 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.889067    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/040f0003-1267-49e6-a365-70608c4bc7d1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.889221 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.889084    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/040f0003-1267-49e6-a365-70608c4bc7d1-secret-grpc-tls\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.889221 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.889143    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/040f0003-1267-49e6-a365-70608c4bc7d1-secret-thanos-querier-tls\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.889221 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.889182    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/040f0003-1267-49e6-a365-70608c4bc7d1-metrics-client-ca\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.889395 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.889259    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25rmc\" (UniqueName: \"kubernetes.io/projected/040f0003-1267-49e6-a365-70608c4bc7d1-kube-api-access-25rmc\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.989760 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.989720    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/040f0003-1267-49e6-a365-70608c4bc7d1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.989760 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.989753    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/040f0003-1267-49e6-a365-70608c4bc7d1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.989760 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.989771    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/040f0003-1267-49e6-a365-70608c4bc7d1-secret-grpc-tls\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.990066 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.989817    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/040f0003-1267-49e6-a365-70608c4bc7d1-secret-thanos-querier-tls\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.990066 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.989849    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/040f0003-1267-49e6-a365-70608c4bc7d1-metrics-client-ca\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.990066 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.989906    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25rmc\" (UniqueName: \"kubernetes.io/projected/040f0003-1267-49e6-a365-70608c4bc7d1-kube-api-access-25rmc\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.990066 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.989963    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/040f0003-1267-49e6-a365-70608c4bc7d1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.990066 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.989992    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/040f0003-1267-49e6-a365-70608c4bc7d1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.991004 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.990977    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/040f0003-1267-49e6-a365-70608c4bc7d1-metrics-client-ca\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.992824 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.992791    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/040f0003-1267-49e6-a365-70608c4bc7d1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.992935 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.992842    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/040f0003-1267-49e6-a365-70608c4bc7d1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.992935 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.992869    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/040f0003-1267-49e6-a365-70608c4bc7d1-secret-thanos-querier-tls\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.993047 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.993001    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/040f0003-1267-49e6-a365-70608c4bc7d1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.993583 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.993558    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/040f0003-1267-49e6-a365-70608c4bc7d1-secret-grpc-tls\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.993964 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.993944    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/040f0003-1267-49e6-a365-70608c4bc7d1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:07.999613 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:07.999593    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25rmc\" (UniqueName: \"kubernetes.io/projected/040f0003-1267-49e6-a365-70608c4bc7d1-kube-api-access-25rmc\") pod \"thanos-querier-b74c4f759-qjbvs\" (UID: \"040f0003-1267-49e6-a365-70608c4bc7d1\") " pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:08.094372 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:08.094327    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:08.294250 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:08.294208    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-96zqr" event={"ID":"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f","Type":"ContainerStarted","Data":"8a28271d0a850d1b75ef154cac9a58ef2a02c102a94bf7b75c7e19fc020566e7"}
Sep 10 15:43:08.294742 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:08.294320    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-96zqr" event={"ID":"ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f","Type":"ContainerStarted","Data":"15e660baed60b00503442de6c4d591976535541fe7a0587b138e38b1c17c66dd"}
Sep 10 15:43:08.318154 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:08.318098    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-96zqr" podStartSLOduration=3.176411732 podStartE2EDuration="4.318081648s" totalImagesPullingTime="1.07384924s" totalInitContainerRuntime="67.820676ms" isStatefulPod=true podCreationTimestamp="2026-09-10 15:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:08.317580574 +0000 UTC m=+60.989990659" watchObservedRunningTime="2026-09-10 15:43:08.318081648 +0000 UTC m=+60.990491763"
Sep 10 15:43:08.373027 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:08.372944    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b74c4f759-qjbvs"]
Sep 10 15:43:08.376837 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:43:08.376807    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040f0003_1267_49e6_a365_70608c4bc7d1.slice/crio-44b95717832b1424d576dc1a7aff422998ed78b0b235957379ae7d85cc5e3de9 WatchSource:0}: Error finding container 44b95717832b1424d576dc1a7aff422998ed78b0b235957379ae7d85cc5e3de9: Status 404 returned error can't find the container with id 44b95717832b1424d576dc1a7aff422998ed78b0b235957379ae7d85cc5e3de9
Sep 10 15:43:09.299446 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:09.299405    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs" event={"ID":"040f0003-1267-49e6-a365-70608c4bc7d1","Type":"ContainerStarted","Data":"44b95717832b1424d576dc1a7aff422998ed78b0b235957379ae7d85cc5e3de9"}
Sep 10 15:43:09.430570 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:09.430533    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-8lm2h"]
Sep 10 15:43:09.432779 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:09.432748    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8lm2h"
Sep 10 15:43:09.434908 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:09.434882    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 10 15:43:09.434908 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:09.434895    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-hptjr\""
Sep 10 15:43:09.443124 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:09.443061    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-8lm2h"]
Sep 10 15:43:09.504283 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:09.504244    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37b85e59-be69-499c-85f4-80822b57748b-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-8lm2h\" (UID: \"37b85e59-be69-499c-85f4-80822b57748b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8lm2h"
Sep 10 15:43:09.504470 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:09.504369    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46nf6\" (UniqueName: \"kubernetes.io/projected/37b85e59-be69-499c-85f4-80822b57748b-kube-api-access-46nf6\") pod \"monitoring-plugin-85bbbbbf4-8lm2h\" (UID: \"37b85e59-be69-499c-85f4-80822b57748b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8lm2h"
Sep 10 15:43:09.605253 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:09.605148    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37b85e59-be69-499c-85f4-80822b57748b-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-8lm2h\" (UID: \"37b85e59-be69-499c-85f4-80822b57748b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8lm2h"
Sep 10 15:43:09.605253 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:09.605213    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46nf6\" (UniqueName: \"kubernetes.io/projected/37b85e59-be69-499c-85f4-80822b57748b-kube-api-access-46nf6\") pod \"monitoring-plugin-85bbbbbf4-8lm2h\" (UID: \"37b85e59-be69-499c-85f4-80822b57748b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8lm2h"
Sep 10 15:43:09.605494 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:43:09.605327    2572 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 10 15:43:09.605494 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:43:09.605406    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37b85e59-be69-499c-85f4-80822b57748b-monitoring-plugin-cert podName:37b85e59-be69-499c-85f4-80822b57748b nodeName:}" failed. No retries permitted until 2026-09-10 15:43:10.105388267 +0000 UTC m=+62.777798328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/37b85e59-be69-499c-85f4-80822b57748b-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-8lm2h" (UID: "37b85e59-be69-499c-85f4-80822b57748b") : secret "monitoring-plugin-cert" not found
Sep 10 15:43:09.616454 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:09.616413    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46nf6\" (UniqueName: \"kubernetes.io/projected/37b85e59-be69-499c-85f4-80822b57748b-kube-api-access-46nf6\") pod \"monitoring-plugin-85bbbbbf4-8lm2h\" (UID: \"37b85e59-be69-499c-85f4-80822b57748b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8lm2h"
Sep 10 15:43:10.108501 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:10.108457    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37b85e59-be69-499c-85f4-80822b57748b-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-8lm2h\" (UID: \"37b85e59-be69-499c-85f4-80822b57748b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8lm2h"
Sep 10 15:43:10.108687 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:43:10.108632    2572 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 10 15:43:10.108738 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:43:10.108708    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37b85e59-be69-499c-85f4-80822b57748b-monitoring-plugin-cert podName:37b85e59-be69-499c-85f4-80822b57748b nodeName:}" failed. No retries permitted until 2026-09-10 15:43:11.108689281 +0000 UTC m=+63.781099342 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/37b85e59-be69-499c-85f4-80822b57748b-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-8lm2h" (UID: "37b85e59-be69-499c-85f4-80822b57748b") : secret "monitoring-plugin-cert" not found
Sep 10 15:43:10.264371 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:10.264335    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xrtrf"
Sep 10 15:43:11.116587 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:11.116047    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37b85e59-be69-499c-85f4-80822b57748b-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-8lm2h\" (UID: \"37b85e59-be69-499c-85f4-80822b57748b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8lm2h"
Sep 10 15:43:11.119364 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:11.119332    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37b85e59-be69-499c-85f4-80822b57748b-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-8lm2h\" (UID: \"37b85e59-be69-499c-85f4-80822b57748b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8lm2h"
Sep 10 15:43:11.244917 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:11.244812    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8lm2h"
Sep 10 15:43:11.310087 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:11.310049    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs" event={"ID":"040f0003-1267-49e6-a365-70608c4bc7d1","Type":"ContainerStarted","Data":"aa50b7ac7721f63847d5d7e7ace9af8659805b8ad7f3ed4ec0e3f8e428c3d174"}
Sep 10 15:43:11.310251 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:11.310159    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs" event={"ID":"040f0003-1267-49e6-a365-70608c4bc7d1","Type":"ContainerStarted","Data":"1ae0ba2db999cfa7db272f250d8e41010a1079a5d3539f4286d7f481b2bf8cba"}
Sep 10 15:43:11.310251 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:11.310205    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs" event={"ID":"040f0003-1267-49e6-a365-70608c4bc7d1","Type":"ContainerStarted","Data":"63b7b2a041540b556da42065a174c50149eebb8f05727c072b6d6ff5ad7e553e"}
Sep 10 15:43:11.540901 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:11.540813    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-8lm2h"]
Sep 10 15:43:11.543736 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:43:11.543706    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b85e59_be69_499c_85f4_80822b57748b.slice/crio-7906de79100c95d397bfcfb9415d7aa0fe4e2f7549260f6b1f00d8c11bd6bbee WatchSource:0}: Error finding container 7906de79100c95d397bfcfb9415d7aa0fe4e2f7549260f6b1f00d8c11bd6bbee: Status 404 returned error can't find the container with id 7906de79100c95d397bfcfb9415d7aa0fe4e2f7549260f6b1f00d8c11bd6bbee
Sep 10 15:43:12.314930 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:12.314891    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8lm2h" event={"ID":"37b85e59-be69-499c-85f4-80822b57748b","Type":"ContainerStarted","Data":"7906de79100c95d397bfcfb9415d7aa0fe4e2f7549260f6b1f00d8c11bd6bbee"}
Sep 10 15:43:12.529509 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:12.529463    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r5qf\" (UniqueName: \"kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf\") pod \"network-check-target-jj8fc\" (UID: \"719ee58f-5767-4813-9b76-0ab62b2c8f3a\") " pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:43:12.529707 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:12.529541    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs\") pod \"network-metrics-daemon-tc4ns\" (UID: \"bde0d086-b64d-4188-b17c-4a69af69c9a5\") " pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:43:12.532389 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:12.532361    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 15:43:12.532525 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:12.532363    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 15:43:12.542557 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:12.542527    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 15:43:12.542807 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:12.542786    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde0d086-b64d-4188-b17c-4a69af69c9a5-metrics-certs\") pod \"network-metrics-daemon-tc4ns\" (UID: \"bde0d086-b64d-4188-b17c-4a69af69c9a5\") " pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:43:12.553116 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:12.553084    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r5qf\" (UniqueName: \"kubernetes.io/projected/719ee58f-5767-4813-9b76-0ab62b2c8f3a-kube-api-access-5r5qf\") pod \"network-check-target-jj8fc\" (UID: \"719ee58f-5767-4813-9b76-0ab62b2c8f3a\") " pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:43:12.598866 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:12.598786    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sjjtv\""
Sep 10 15:43:12.606660 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:12.606631    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:43:12.609036 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:12.609009    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2gdj6\""
Sep 10 15:43:12.616813 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:12.616786    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc4ns"
Sep 10 15:43:12.631016 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:12.630983    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret\") pod \"global-pull-secret-syncer-vz9z7\" (UID: \"a7d7c20f-a31c-4468-a822-a7e2601fe3e4\") " pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:43:12.633517 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:12.633495    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 15:43:12.644107 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:12.644071    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7d7c20f-a31c-4468-a822-a7e2601fe3e4-original-pull-secret\") pod \"global-pull-secret-syncer-vz9z7\" (UID: \"a7d7c20f-a31c-4468-a822-a7e2601fe3e4\") " pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:43:12.902159 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:12.902130    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vz9z7"
Sep 10 15:43:13.032599 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:13.032566    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tc4ns"]
Sep 10 15:43:13.035325 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:13.035281    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jj8fc"]
Sep 10 15:43:13.036117 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:43:13.036096    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde0d086_b64d_4188_b17c_4a69af69c9a5.slice/crio-2ae02b3bd193df32cbce50684df0e08cb869b659cbfc3629d4e1c217273698cb WatchSource:0}: Error finding container 2ae02b3bd193df32cbce50684df0e08cb869b659cbfc3629d4e1c217273698cb: Status 404 returned error can't find the container with id 2ae02b3bd193df32cbce50684df0e08cb869b659cbfc3629d4e1c217273698cb
Sep 10 15:43:13.038603 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:43:13.038572    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719ee58f_5767_4813_9b76_0ab62b2c8f3a.slice/crio-ec2c79c8a9f9f892ec1c06b1ba438df685d3fddd95242ad3099674897cd512dd WatchSource:0}: Error finding container ec2c79c8a9f9f892ec1c06b1ba438df685d3fddd95242ad3099674897cd512dd: Status 404 returned error can't find the container with id ec2c79c8a9f9f892ec1c06b1ba438df685d3fddd95242ad3099674897cd512dd
Sep 10 15:43:13.176510 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:13.176430    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vz9z7"]
Sep 10 15:43:13.179763 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:43:13.179737    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d7c20f_a31c_4468_a822_a7e2601fe3e4.slice/crio-1901d1507bd02b48912f2d1f76eed3db17863a462f15f2f55b11c483eea619b0 WatchSource:0}: Error finding container 1901d1507bd02b48912f2d1f76eed3db17863a462f15f2f55b11c483eea619b0: Status 404 returned error can't find the container with id 1901d1507bd02b48912f2d1f76eed3db17863a462f15f2f55b11c483eea619b0
Sep 10 15:43:13.320143 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:13.320091    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tc4ns" event={"ID":"bde0d086-b64d-4188-b17c-4a69af69c9a5","Type":"ContainerStarted","Data":"2ae02b3bd193df32cbce50684df0e08cb869b659cbfc3629d4e1c217273698cb"}
Sep 10 15:43:13.323123 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:13.323095    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs" event={"ID":"040f0003-1267-49e6-a365-70608c4bc7d1","Type":"ContainerStarted","Data":"19ef7124b1aa99515a207fe34b168615bf473cc1517e1caa91387abb4d0f9f15"}
Sep 10 15:43:13.323289 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:13.323152    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs" event={"ID":"040f0003-1267-49e6-a365-70608c4bc7d1","Type":"ContainerStarted","Data":"278adfe1a4c2774b64b1816f83bd589afbf1ad13788d7ae929943c123f876d1c"}
Sep 10 15:43:13.323289 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:13.323178    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs" event={"ID":"040f0003-1267-49e6-a365-70608c4bc7d1","Type":"ContainerStarted","Data":"6c0d6c0aa6bc8540d41b39a9ffb8418e733e6c9e8cc291988be74e013dbf2b95"}
Sep 10 15:43:13.323289 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:13.323240    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:13.324747 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:13.324711    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8lm2h" event={"ID":"37b85e59-be69-499c-85f4-80822b57748b","Type":"ContainerStarted","Data":"cebd846072d7cc1bdbf77fe6819a6b0552fd96d4310d43457288c9cae91abae0"}
Sep 10 15:43:13.324847 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:13.324760    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8lm2h"
Sep 10 15:43:13.325984 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:13.325964    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vz9z7" event={"ID":"a7d7c20f-a31c-4468-a822-a7e2601fe3e4","Type":"ContainerStarted","Data":"1901d1507bd02b48912f2d1f76eed3db17863a462f15f2f55b11c483eea619b0"}
Sep 10 15:43:13.327413 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:13.327384    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jj8fc" event={"ID":"719ee58f-5767-4813-9b76-0ab62b2c8f3a","Type":"ContainerStarted","Data":"ec2c79c8a9f9f892ec1c06b1ba438df685d3fddd95242ad3099674897cd512dd"}
Sep 10 15:43:13.329952 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:13.329933    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8lm2h"
Sep 10 15:43:13.344517 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:13.344411    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs" podStartSLOduration=2.313775361 podStartE2EDuration="6.344389266s" totalImagesPullingTime="4.030613905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:13.34346932 +0000 UTC m=+66.015879403" watchObservedRunningTime="2026-09-10 15:43:13.344389266 +0000 UTC m=+66.016799349"
Sep 10 15:43:13.360028 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:13.359972    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8lm2h" podStartSLOduration=3.120866538 podStartE2EDuration="4.359956915s" totalImagesPullingTime="1.239090377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:13.359272188 +0000 UTC m=+66.031682270" watchObservedRunningTime="2026-09-10 15:43:13.359956915 +0000 UTC m=+66.032366997"
Sep 10 15:43:14.335739 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:14.335695    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tc4ns" event={"ID":"bde0d086-b64d-4188-b17c-4a69af69c9a5","Type":"ContainerStarted","Data":"c8ad5dcc439303554da819173803c3b34d1ad47d84ee6b7610a5a50a5d3190be"}
Sep 10 15:43:14.336205 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:14.335767    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tc4ns" event={"ID":"bde0d086-b64d-4188-b17c-4a69af69c9a5","Type":"ContainerStarted","Data":"a3b0fb6b2a745b308efd0e3ca5990ff44539f1ea57ba32fcaf9223357b0c331f"}
Sep 10 15:43:14.353069 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:14.352830    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tc4ns" podStartSLOduration=66.287462514 podStartE2EDuration="1m7.352797606s" totalImagesPullingTime="1.065335092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:14.350860619 +0000 UTC m=+67.023270702" watchObservedRunningTime="2026-09-10 15:43:14.352797606 +0000 UTC m=+67.025207891"
Sep 10 15:43:18.249214 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:18.249184    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d854b8bbc-pv7h9"
Sep 10 15:43:18.357105 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:18.357061    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vz9z7" event={"ID":"a7d7c20f-a31c-4468-a822-a7e2601fe3e4","Type":"ContainerStarted","Data":"be7c75999e11859b21f78496925e0778d0a2890ec00bb07e1707a85ae944653e"}
Sep 10 15:43:18.358756 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:18.358729    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jj8fc" event={"ID":"719ee58f-5767-4813-9b76-0ab62b2c8f3a","Type":"ContainerStarted","Data":"9e09d338cb0117920e8734f9603c49771f00fef20e27440343d8b160e331659c"}
Sep 10 15:43:18.358885 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:18.358857    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:43:18.417061 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:18.416921    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vz9z7" podStartSLOduration=66.0904097 podStartE2EDuration="1m10.416900753s" totalImagesPullingTime="4.326491053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:18.416008039 +0000 UTC m=+71.088418136" watchObservedRunningTime="2026-09-10 15:43:18.416900753 +0000 UTC m=+71.089310837"
Sep 10 15:43:18.441425 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:18.441364    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jj8fc" podStartSLOduration=66.978045365 podStartE2EDuration="1m11.441344615s" totalImagesPullingTime="4.46329925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:18.440789887 +0000 UTC m=+71.113199969" watchObservedRunningTime="2026-09-10 15:43:18.441344615 +0000 UTC m=+71.113754699"
Sep 10 15:43:19.341570 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:19.341541    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b74c4f759-qjbvs"
Sep 10 15:43:49.365225 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:43:49.365189    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jj8fc"
Sep 10 15:44:26.980971 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:26.980936    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 15:44:26.984546 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:26.984524    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:26.986655 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:26.986632    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 15:44:26.986786 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:26.986676    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 15:44:26.986786 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:26.986634    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 15:44:26.987029 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:26.986973    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 15:44:26.987029 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:26.986973    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-96vnm\""
Sep 10 15:44:26.987221 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:26.987191    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 15:44:26.987221 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:26.987212    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 15:44:26.987369 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:26.987263    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 15:44:26.987369 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:26.987270    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 15:44:26.992531 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:26.992509    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 15:44:26.998705 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:26.998675    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 15:44:27.087854 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.087805    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.087854 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.087858    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e14d9389-fb0b-4162-84d9-4a128d7a95c6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.088103 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.087887    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e14d9389-fb0b-4162-84d9-4a128d7a95c6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.088103 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.087913    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.088103 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.087939    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.088103 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.087982    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-config-volume\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.088103 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.088024    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e14d9389-fb0b-4162-84d9-4a128d7a95c6-config-out\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.088103 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.088103    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.088443 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.088140    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.088443 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.088216    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e14d9389-fb0b-4162-84d9-4a128d7a95c6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.088443 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.088334    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-web-config\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.088443 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.088367    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65mwg\" (UniqueName: \"kubernetes.io/projected/e14d9389-fb0b-4162-84d9-4a128d7a95c6-kube-api-access-65mwg\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.088443 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.088410    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e14d9389-fb0b-4162-84d9-4a128d7a95c6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.189390 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.189342    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.189390 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.189398    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e14d9389-fb0b-4162-84d9-4a128d7a95c6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.189552 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.189419    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e14d9389-fb0b-4162-84d9-4a128d7a95c6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.189552 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.189439    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.189552 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.189458    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.189552 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.189474    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-config-volume\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.189552 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.189528    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e14d9389-fb0b-4162-84d9-4a128d7a95c6-config-out\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.189796 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.189588    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.189796 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.189617    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.189796 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.189695    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e14d9389-fb0b-4162-84d9-4a128d7a95c6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.189796 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.189725    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-web-config\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.189796 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.189754    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65mwg\" (UniqueName: \"kubernetes.io/projected/e14d9389-fb0b-4162-84d9-4a128d7a95c6-kube-api-access-65mwg\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.189796 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.189784    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e14d9389-fb0b-4162-84d9-4a128d7a95c6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.190081 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.189913    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e14d9389-fb0b-4162-84d9-4a128d7a95c6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.190820 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.190789    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e14d9389-fb0b-4162-84d9-4a128d7a95c6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.191950 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.191922    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e14d9389-fb0b-4162-84d9-4a128d7a95c6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.192629 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.192593    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.192629 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.192598    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e14d9389-fb0b-4162-84d9-4a128d7a95c6-config-out\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.192797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.192647    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-config-volume\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.192797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.192707    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e14d9389-fb0b-4162-84d9-4a128d7a95c6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.192797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.192765    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.193284 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.193261    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.193522 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.193499    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.194593 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.194571    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-web-config\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.194716 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.194682    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e14d9389-fb0b-4162-84d9-4a128d7a95c6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.200202 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.200179    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65mwg\" (UniqueName: \"kubernetes.io/projected/e14d9389-fb0b-4162-84d9-4a128d7a95c6-kube-api-access-65mwg\") pod \"alertmanager-main-0\" (UID: \"e14d9389-fb0b-4162-84d9-4a128d7a95c6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.293975 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.293865    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 15:44:27.572550 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.572428    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 15:44:27.578783 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:44:27.578751    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14d9389_fb0b_4162_84d9_4a128d7a95c6.slice/crio-4e161bfdf18474b0892aa5107d94226318ee1c86fb91996a871b6bfdbf63f076 WatchSource:0}: Error finding container 4e161bfdf18474b0892aa5107d94226318ee1c86fb91996a871b6bfdbf63f076: Status 404 returned error can't find the container with id 4e161bfdf18474b0892aa5107d94226318ee1c86fb91996a871b6bfdbf63f076
Sep 10 15:44:27.601423 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:27.601381    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e14d9389-fb0b-4162-84d9-4a128d7a95c6","Type":"ContainerStarted","Data":"4e161bfdf18474b0892aa5107d94226318ee1c86fb91996a871b6bfdbf63f076"}
Sep 10 15:44:29.610643 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:29.610605    2572 generic.go:356] "Generic (PLEG): container finished" podID="e14d9389-fb0b-4162-84d9-4a128d7a95c6" containerID="f6841bc412a980f3d590528a08eca7057acde0864030cb0174886f724f00aa3a" exitCode=0
Sep 10 15:44:29.611073 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:29.610673    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e14d9389-fb0b-4162-84d9-4a128d7a95c6","Type":"ContainerDied","Data":"f6841bc412a980f3d590528a08eca7057acde0864030cb0174886f724f00aa3a"}
Sep 10 15:44:31.621841 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:31.621805    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e14d9389-fb0b-4162-84d9-4a128d7a95c6","Type":"ContainerStarted","Data":"38d57b5c09ef1c4ccd829636bbdab89171cf8930ef2c592ee1dce906793de26a"}
Sep 10 15:44:31.622317 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:31.621922    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e14d9389-fb0b-4162-84d9-4a128d7a95c6","Type":"ContainerStarted","Data":"c9585603e0c39ab130692ff24c42f7b184b228ce1f92c181f3aa41bc2d71d667"}
Sep 10 15:44:31.622317 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:31.621956    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e14d9389-fb0b-4162-84d9-4a128d7a95c6","Type":"ContainerStarted","Data":"9c8467c8a5b86ca50ca9bef1fa0df45e2e360559f9055d74b8695e82798ab3ef"}
Sep 10 15:44:31.622317 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:31.621980    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e14d9389-fb0b-4162-84d9-4a128d7a95c6","Type":"ContainerStarted","Data":"67d69e72625b8713bd91949ecc17c962540dd9d85b0dc2d127ac8cf637d3f355"}
Sep 10 15:44:31.622317 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:31.622003    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e14d9389-fb0b-4162-84d9-4a128d7a95c6","Type":"ContainerStarted","Data":"55bbbf33c9433880c4f793d41b48c82a395358e6df10d7cf2d0a5a36353106ee"}
Sep 10 15:44:31.622317 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:31.622024    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e14d9389-fb0b-4162-84d9-4a128d7a95c6","Type":"ContainerStarted","Data":"426ce7dafed43d9c9bb1c5b99473ef3c1a71943aebc2a0ea4282f5fc9a7cee74"}
Sep 10 15:44:31.648771 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:44:31.648701    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.324096429 podStartE2EDuration="5.648680193s" totalImagesPullingTime="2.30669899s" totalInitContainerRuntime="17.884774ms" isStatefulPod=false podCreationTimestamp="2026-09-10 15:44:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:31.64746029 +0000 UTC m=+144.319870372" watchObservedRunningTime="2026-09-10 15:44:31.648680193 +0000 UTC m=+144.321090277"
Sep 10 15:46:25.505126 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:25.505089    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8"]
Sep 10 15:46:25.507443 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:25.507425    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8"
Sep 10 15:46:25.509730 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:25.509710    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 15:46:25.509838 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:25.509712    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hg86r\""
Sep 10 15:46:25.510858 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:25.510839    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 15:46:25.521262 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:25.521236    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8"]
Sep 10 15:46:25.544645 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:25.544612    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a0fec9a-ddd8-447d-83d2-6e460e734b70-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8\" (UID: \"5a0fec9a-ddd8-447d-83d2-6e460e734b70\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8"
Sep 10 15:46:25.544815 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:25.544654    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc9dt\" (UniqueName: \"kubernetes.io/projected/5a0fec9a-ddd8-447d-83d2-6e460e734b70-kube-api-access-nc9dt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8\" (UID: \"5a0fec9a-ddd8-447d-83d2-6e460e734b70\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8"
Sep 10 15:46:25.544815 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:25.544685    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a0fec9a-ddd8-447d-83d2-6e460e734b70-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8\" (UID: \"5a0fec9a-ddd8-447d-83d2-6e460e734b70\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8"
Sep 10 15:46:25.645418 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:25.645383    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a0fec9a-ddd8-447d-83d2-6e460e734b70-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8\" (UID: \"5a0fec9a-ddd8-447d-83d2-6e460e734b70\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8"
Sep 10 15:46:25.645580 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:25.645433    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc9dt\" (UniqueName: \"kubernetes.io/projected/5a0fec9a-ddd8-447d-83d2-6e460e734b70-kube-api-access-nc9dt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8\" (UID: \"5a0fec9a-ddd8-447d-83d2-6e460e734b70\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8"
Sep 10 15:46:25.645580 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:25.645467    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a0fec9a-ddd8-447d-83d2-6e460e734b70-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8\" (UID: \"5a0fec9a-ddd8-447d-83d2-6e460e734b70\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8"
Sep 10 15:46:25.645828 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:25.645804    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a0fec9a-ddd8-447d-83d2-6e460e734b70-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8\" (UID: \"5a0fec9a-ddd8-447d-83d2-6e460e734b70\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8"
Sep 10 15:46:25.645828 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:25.645821    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a0fec9a-ddd8-447d-83d2-6e460e734b70-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8\" (UID: \"5a0fec9a-ddd8-447d-83d2-6e460e734b70\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8"
Sep 10 15:46:25.654853 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:25.654833    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc9dt\" (UniqueName: \"kubernetes.io/projected/5a0fec9a-ddd8-447d-83d2-6e460e734b70-kube-api-access-nc9dt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8\" (UID: \"5a0fec9a-ddd8-447d-83d2-6e460e734b70\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8"
Sep 10 15:46:25.816752 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:25.816720    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8"
Sep 10 15:46:26.084564 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:26.084538    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8"]
Sep 10 15:46:26.086901 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:46:26.086871    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0fec9a_ddd8_447d_83d2_6e460e734b70.slice/crio-5284f87c16dab26433095c33d9c8fa5c08ecb378a535047a0a53ed06b1a06b46 WatchSource:0}: Error finding container 5284f87c16dab26433095c33d9c8fa5c08ecb378a535047a0a53ed06b1a06b46: Status 404 returned error can't find the container with id 5284f87c16dab26433095c33d9c8fa5c08ecb378a535047a0a53ed06b1a06b46
Sep 10 15:46:27.037814 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:27.037773    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8" event={"ID":"5a0fec9a-ddd8-447d-83d2-6e460e734b70","Type":"ContainerStarted","Data":"5284f87c16dab26433095c33d9c8fa5c08ecb378a535047a0a53ed06b1a06b46"}
Sep 10 15:46:32.059810 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:32.059770    2572 generic.go:356] "Generic (PLEG): container finished" podID="5a0fec9a-ddd8-447d-83d2-6e460e734b70" containerID="a44e7c6037fb564f72c6db9987469f54f5ec26180477743084c3146a19987236" exitCode=0
Sep 10 15:46:32.060268 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:32.059841    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8" event={"ID":"5a0fec9a-ddd8-447d-83d2-6e460e734b70","Type":"ContainerDied","Data":"a44e7c6037fb564f72c6db9987469f54f5ec26180477743084c3146a19987236"}
Sep 10 15:46:34.069808 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:34.069709    2572 generic.go:356] "Generic (PLEG): container finished" podID="5a0fec9a-ddd8-447d-83d2-6e460e734b70" containerID="675af7a39bb665d87959c185b739d08703e526649e8e484dc58fa593d1754320" exitCode=0
Sep 10 15:46:34.070176 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:34.069796    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8" event={"ID":"5a0fec9a-ddd8-447d-83d2-6e460e734b70","Type":"ContainerDied","Data":"675af7a39bb665d87959c185b739d08703e526649e8e484dc58fa593d1754320"}
Sep 10 15:46:41.099826 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:41.099788    2572 generic.go:356] "Generic (PLEG): container finished" podID="5a0fec9a-ddd8-447d-83d2-6e460e734b70" containerID="89eee4c0388e55694f4e304ae96c9c8bf5a4629ccc14954466f0999fa71354cf" exitCode=0
Sep 10 15:46:41.100222 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:41.099874    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8" event={"ID":"5a0fec9a-ddd8-447d-83d2-6e460e734b70","Type":"ContainerDied","Data":"89eee4c0388e55694f4e304ae96c9c8bf5a4629ccc14954466f0999fa71354cf"}
Sep 10 15:46:42.227253 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:42.227224    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8"
Sep 10 15:46:42.289000 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:42.288960    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nc9dt\" (UniqueName: \"kubernetes.io/projected/5a0fec9a-ddd8-447d-83d2-6e460e734b70-kube-api-access-nc9dt\") pod \"5a0fec9a-ddd8-447d-83d2-6e460e734b70\" (UID: \"5a0fec9a-ddd8-447d-83d2-6e460e734b70\") "
Sep 10 15:46:42.289168 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:42.289018    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a0fec9a-ddd8-447d-83d2-6e460e734b70-util\") pod \"5a0fec9a-ddd8-447d-83d2-6e460e734b70\" (UID: \"5a0fec9a-ddd8-447d-83d2-6e460e734b70\") "
Sep 10 15:46:42.289168 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:42.289038    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a0fec9a-ddd8-447d-83d2-6e460e734b70-bundle\") pod \"5a0fec9a-ddd8-447d-83d2-6e460e734b70\" (UID: \"5a0fec9a-ddd8-447d-83d2-6e460e734b70\") "
Sep 10 15:46:42.289690 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:42.289662    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a0fec9a-ddd8-447d-83d2-6e460e734b70-bundle" (OuterVolumeSpecName: "bundle") pod "5a0fec9a-ddd8-447d-83d2-6e460e734b70" (UID: "5a0fec9a-ddd8-447d-83d2-6e460e734b70"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:46:42.291333 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:42.291283    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0fec9a-ddd8-447d-83d2-6e460e734b70-kube-api-access-nc9dt" (OuterVolumeSpecName: "kube-api-access-nc9dt") pod "5a0fec9a-ddd8-447d-83d2-6e460e734b70" (UID: "5a0fec9a-ddd8-447d-83d2-6e460e734b70"). InnerVolumeSpecName "kube-api-access-nc9dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:46:42.292906 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:42.292880    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a0fec9a-ddd8-447d-83d2-6e460e734b70-util" (OuterVolumeSpecName: "util") pod "5a0fec9a-ddd8-447d-83d2-6e460e734b70" (UID: "5a0fec9a-ddd8-447d-83d2-6e460e734b70"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:46:42.390660 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:42.390558    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a0fec9a-ddd8-447d-83d2-6e460e734b70-util\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:46:42.390660 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:42.390603    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a0fec9a-ddd8-447d-83d2-6e460e734b70-bundle\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:46:42.390660 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:42.390613    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc9dt\" (UniqueName: \"kubernetes.io/projected/5a0fec9a-ddd8-447d-83d2-6e460e734b70-kube-api-access-nc9dt\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:46:43.109086 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:43.109047    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8" event={"ID":"5a0fec9a-ddd8-447d-83d2-6e460e734b70","Type":"ContainerDied","Data":"5284f87c16dab26433095c33d9c8fa5c08ecb378a535047a0a53ed06b1a06b46"}
Sep 10 15:46:43.109086 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:43.109071    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409gmv8"
Sep 10 15:46:43.109315 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:43.109105    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5284f87c16dab26433095c33d9c8fa5c08ecb378a535047a0a53ed06b1a06b46"
Sep 10 15:46:48.798567 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:48.798528    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkkxt"]
Sep 10 15:46:48.798961 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:48.798821    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a0fec9a-ddd8-447d-83d2-6e460e734b70" containerName="util"
Sep 10 15:46:48.798961 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:48.798833    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a0fec9a-ddd8-447d-83d2-6e460e734b70" containerName="util"
Sep 10 15:46:48.798961 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:48.798844    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a0fec9a-ddd8-447d-83d2-6e460e734b70" containerName="extract"
Sep 10 15:46:48.798961 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:48.798850    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a0fec9a-ddd8-447d-83d2-6e460e734b70" containerName="extract"
Sep 10 15:46:48.798961 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:48.798864    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a0fec9a-ddd8-447d-83d2-6e460e734b70" containerName="pull"
Sep 10 15:46:48.798961 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:48.798869    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a0fec9a-ddd8-447d-83d2-6e460e734b70" containerName="pull"
Sep 10 15:46:48.798961 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:48.798914    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a0fec9a-ddd8-447d-83d2-6e460e734b70" containerName="extract"
Sep 10 15:46:48.801944 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:48.801926    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkkxt"
Sep 10 15:46:48.804270 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:48.804226    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:46:48.804773 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:48.804757    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-888k4\""
Sep 10 15:46:48.804844 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:48.804814    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 10 15:46:48.819531 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:48.819502    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkkxt"]
Sep 10 15:46:48.944226 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:48.944179    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9017df4a-b541-4a0e-b30f-3f693a97bea1-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jkkxt\" (UID: \"9017df4a-b541-4a0e-b30f-3f693a97bea1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkkxt"
Sep 10 15:46:48.944427 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:48.944246    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds47k\" (UniqueName: \"kubernetes.io/projected/9017df4a-b541-4a0e-b30f-3f693a97bea1-kube-api-access-ds47k\") pod \"cert-manager-operator-controller-manager-55c444bf86-jkkxt\" (UID: \"9017df4a-b541-4a0e-b30f-3f693a97bea1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkkxt"
Sep 10 15:46:49.045668 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:49.045619    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9017df4a-b541-4a0e-b30f-3f693a97bea1-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jkkxt\" (UID: \"9017df4a-b541-4a0e-b30f-3f693a97bea1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkkxt"
Sep 10 15:46:49.045810 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:49.045686    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds47k\" (UniqueName: \"kubernetes.io/projected/9017df4a-b541-4a0e-b30f-3f693a97bea1-kube-api-access-ds47k\") pod \"cert-manager-operator-controller-manager-55c444bf86-jkkxt\" (UID: \"9017df4a-b541-4a0e-b30f-3f693a97bea1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkkxt"
Sep 10 15:46:49.046058 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:49.046032    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9017df4a-b541-4a0e-b30f-3f693a97bea1-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jkkxt\" (UID: \"9017df4a-b541-4a0e-b30f-3f693a97bea1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkkxt"
Sep 10 15:46:49.056687 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:49.056629    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds47k\" (UniqueName: \"kubernetes.io/projected/9017df4a-b541-4a0e-b30f-3f693a97bea1-kube-api-access-ds47k\") pod \"cert-manager-operator-controller-manager-55c444bf86-jkkxt\" (UID: \"9017df4a-b541-4a0e-b30f-3f693a97bea1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkkxt"
Sep 10 15:46:49.110625 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:49.110583    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkkxt"
Sep 10 15:46:49.391118 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:49.391030    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkkxt"]
Sep 10 15:46:49.394375 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:46:49.394326    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9017df4a_b541_4a0e_b30f_3f693a97bea1.slice/crio-bb02cf5fc282f55ab085e6555dfe635e4db31695a327b9aa5f9b755676c83258 WatchSource:0}: Error finding container bb02cf5fc282f55ab085e6555dfe635e4db31695a327b9aa5f9b755676c83258: Status 404 returned error can't find the container with id bb02cf5fc282f55ab085e6555dfe635e4db31695a327b9aa5f9b755676c83258
Sep 10 15:46:50.136391 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:50.136353    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkkxt" event={"ID":"9017df4a-b541-4a0e-b30f-3f693a97bea1","Type":"ContainerStarted","Data":"bb02cf5fc282f55ab085e6555dfe635e4db31695a327b9aa5f9b755676c83258"}
Sep 10 15:46:52.147778 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:52.147735    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkkxt" event={"ID":"9017df4a-b541-4a0e-b30f-3f693a97bea1","Type":"ContainerStarted","Data":"a7fc05aae8463b8c33ec5defa14c00bf5cf5fe3d32125e2a1098213b4dbc5322"}
Sep 10 15:46:52.209430 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:52.209374    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jkkxt" podStartSLOduration=2.031314824 podStartE2EDuration="4.209357718s" totalImagesPullingTime="2.178042894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:52.208690886 +0000 UTC m=+284.881100982" watchObservedRunningTime="2026-09-10 15:46:52.209357718 +0000 UTC m=+284.881767802"
Sep 10 15:46:53.403841 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:53.403794    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd"]
Sep 10 15:46:53.407177 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:53.407155    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd"
Sep 10 15:46:53.409991 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:53.409966    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 15:46:53.410132 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:53.410065    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 15:46:53.411087 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:53.411067    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hg86r\""
Sep 10 15:46:53.423430 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:53.423396    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd"]
Sep 10 15:46:53.479713 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:53.479675    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4940696-5525-43f0-b39b-7d3cdd5518a6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd\" (UID: \"b4940696-5525-43f0-b39b-7d3cdd5518a6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd"
Sep 10 15:46:53.479912 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:53.479737    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4940696-5525-43f0-b39b-7d3cdd5518a6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd\" (UID: \"b4940696-5525-43f0-b39b-7d3cdd5518a6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd"
Sep 10 15:46:53.479912 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:53.479815    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx7ds\" (UniqueName: \"kubernetes.io/projected/b4940696-5525-43f0-b39b-7d3cdd5518a6-kube-api-access-gx7ds\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd\" (UID: \"b4940696-5525-43f0-b39b-7d3cdd5518a6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd"
Sep 10 15:46:53.581033 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:53.580989    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4940696-5525-43f0-b39b-7d3cdd5518a6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd\" (UID: \"b4940696-5525-43f0-b39b-7d3cdd5518a6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd"
Sep 10 15:46:53.581207 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:53.581045    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4940696-5525-43f0-b39b-7d3cdd5518a6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd\" (UID: \"b4940696-5525-43f0-b39b-7d3cdd5518a6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd"
Sep 10 15:46:53.581207 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:53.581075    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx7ds\" (UniqueName: \"kubernetes.io/projected/b4940696-5525-43f0-b39b-7d3cdd5518a6-kube-api-access-gx7ds\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd\" (UID: \"b4940696-5525-43f0-b39b-7d3cdd5518a6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd"
Sep 10 15:46:53.581569 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:53.581544    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4940696-5525-43f0-b39b-7d3cdd5518a6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd\" (UID: \"b4940696-5525-43f0-b39b-7d3cdd5518a6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd"
Sep 10 15:46:53.581615 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:53.581544    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4940696-5525-43f0-b39b-7d3cdd5518a6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd\" (UID: \"b4940696-5525-43f0-b39b-7d3cdd5518a6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd"
Sep 10 15:46:53.593361 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:53.593329    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx7ds\" (UniqueName: \"kubernetes.io/projected/b4940696-5525-43f0-b39b-7d3cdd5518a6-kube-api-access-gx7ds\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd\" (UID: \"b4940696-5525-43f0-b39b-7d3cdd5518a6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd"
Sep 10 15:46:53.717129 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:53.717018    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd"
Sep 10 15:46:53.993873 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:53.992977    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd"]
Sep 10 15:46:53.997717 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:46:53.997680    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4940696_5525_43f0_b39b_7d3cdd5518a6.slice/crio-12508df7d300cfab40dafadc72e97144b45b2d433e44419046dac80d93239569 WatchSource:0}: Error finding container 12508df7d300cfab40dafadc72e97144b45b2d433e44419046dac80d93239569: Status 404 returned error can't find the container with id 12508df7d300cfab40dafadc72e97144b45b2d433e44419046dac80d93239569
Sep 10 15:46:54.156704 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:54.156665    2572 generic.go:356] "Generic (PLEG): container finished" podID="b4940696-5525-43f0-b39b-7d3cdd5518a6" containerID="42fca3417b43a0a4e22356f02448df0f38c1bebfaed1abdd5a6039821457ccba" exitCode=0
Sep 10 15:46:54.156893 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:54.156740    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd" event={"ID":"b4940696-5525-43f0-b39b-7d3cdd5518a6","Type":"ContainerDied","Data":"42fca3417b43a0a4e22356f02448df0f38c1bebfaed1abdd5a6039821457ccba"}
Sep 10 15:46:54.156893 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:54.156796    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd" event={"ID":"b4940696-5525-43f0-b39b-7d3cdd5518a6","Type":"ContainerStarted","Data":"12508df7d300cfab40dafadc72e97144b45b2d433e44419046dac80d93239569"}
Sep 10 15:46:56.171009 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:56.170968    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd" event={"ID":"b4940696-5525-43f0-b39b-7d3cdd5518a6","Type":"ContainerStarted","Data":"2945083e7426d19c8a7deea2b45da93a4e5e77f866d6ed67ed82478ed7f4add8"}
Sep 10 15:46:57.176677 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:57.176636    2572 generic.go:356] "Generic (PLEG): container finished" podID="b4940696-5525-43f0-b39b-7d3cdd5518a6" containerID="2945083e7426d19c8a7deea2b45da93a4e5e77f866d6ed67ed82478ed7f4add8" exitCode=0
Sep 10 15:46:57.177070 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:57.176718    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd" event={"ID":"b4940696-5525-43f0-b39b-7d3cdd5518a6","Type":"ContainerDied","Data":"2945083e7426d19c8a7deea2b45da93a4e5e77f866d6ed67ed82478ed7f4add8"}
Sep 10 15:46:58.183552 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:58.183515    2572 generic.go:356] "Generic (PLEG): container finished" podID="b4940696-5525-43f0-b39b-7d3cdd5518a6" containerID="aecb65fa1766d95f9502d03889fb9e13b4bcbf43d311a3579f3283eac15dc300" exitCode=0
Sep 10 15:46:58.184031 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:58.183606    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd" event={"ID":"b4940696-5525-43f0-b39b-7d3cdd5518a6","Type":"ContainerDied","Data":"aecb65fa1766d95f9502d03889fb9e13b4bcbf43d311a3579f3283eac15dc300"}
Sep 10 15:46:59.315505 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.315461    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd"
Sep 10 15:46:59.435273 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.435236    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4940696-5525-43f0-b39b-7d3cdd5518a6-bundle\") pod \"b4940696-5525-43f0-b39b-7d3cdd5518a6\" (UID: \"b4940696-5525-43f0-b39b-7d3cdd5518a6\") "
Sep 10 15:46:59.435273 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.435280    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx7ds\" (UniqueName: \"kubernetes.io/projected/b4940696-5525-43f0-b39b-7d3cdd5518a6-kube-api-access-gx7ds\") pod \"b4940696-5525-43f0-b39b-7d3cdd5518a6\" (UID: \"b4940696-5525-43f0-b39b-7d3cdd5518a6\") "
Sep 10 15:46:59.435545 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.435367    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4940696-5525-43f0-b39b-7d3cdd5518a6-util\") pod \"b4940696-5525-43f0-b39b-7d3cdd5518a6\" (UID: \"b4940696-5525-43f0-b39b-7d3cdd5518a6\") "
Sep 10 15:46:59.435765 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.435724    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4940696-5525-43f0-b39b-7d3cdd5518a6-bundle" (OuterVolumeSpecName: "bundle") pod "b4940696-5525-43f0-b39b-7d3cdd5518a6" (UID: "b4940696-5525-43f0-b39b-7d3cdd5518a6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:46:59.437616 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.437588    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4940696-5525-43f0-b39b-7d3cdd5518a6-kube-api-access-gx7ds" (OuterVolumeSpecName: "kube-api-access-gx7ds") pod "b4940696-5525-43f0-b39b-7d3cdd5518a6" (UID: "b4940696-5525-43f0-b39b-7d3cdd5518a6"). InnerVolumeSpecName "kube-api-access-gx7ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:46:59.439812 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.439747    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4940696-5525-43f0-b39b-7d3cdd5518a6-util" (OuterVolumeSpecName: "util") pod "b4940696-5525-43f0-b39b-7d3cdd5518a6" (UID: "b4940696-5525-43f0-b39b-7d3cdd5518a6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:46:59.536292 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.536244    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4940696-5525-43f0-b39b-7d3cdd5518a6-bundle\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:46:59.536292 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.536286    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx7ds\" (UniqueName: \"kubernetes.io/projected/b4940696-5525-43f0-b39b-7d3cdd5518a6-kube-api-access-gx7ds\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:46:59.536292 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.536298    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4940696-5525-43f0-b39b-7d3cdd5518a6-util\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:46:59.660641 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.660605    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-frwx2"]
Sep 10 15:46:59.660905 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.660894    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4940696-5525-43f0-b39b-7d3cdd5518a6" containerName="util"
Sep 10 15:46:59.660961 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.660907    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4940696-5525-43f0-b39b-7d3cdd5518a6" containerName="util"
Sep 10 15:46:59.660961 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.660918    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4940696-5525-43f0-b39b-7d3cdd5518a6" containerName="extract"
Sep 10 15:46:59.660961 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.660924    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4940696-5525-43f0-b39b-7d3cdd5518a6" containerName="extract"
Sep 10 15:46:59.660961 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.660937    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4940696-5525-43f0-b39b-7d3cdd5518a6" containerName="pull"
Sep 10 15:46:59.660961 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.660943    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4940696-5525-43f0-b39b-7d3cdd5518a6" containerName="pull"
Sep 10 15:46:59.661103 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.660986    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4940696-5525-43f0-b39b-7d3cdd5518a6" containerName="extract"
Sep 10 15:46:59.663715 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.663694    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-frwx2"
Sep 10 15:46:59.665958 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.665935    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 15:46:59.666272 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.666250    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-nf5tr\""
Sep 10 15:46:59.666450 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.666319    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 15:46:59.673755 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.673730    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-frwx2"]
Sep 10 15:46:59.738115 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.738011    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35444ca7-f346-4d21-ba2d-6b0aec5ad8ae-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-frwx2\" (UID: \"35444ca7-f346-4d21-ba2d-6b0aec5ad8ae\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-frwx2"
Sep 10 15:46:59.738115 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.738078    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd6rs\" (UniqueName: \"kubernetes.io/projected/35444ca7-f346-4d21-ba2d-6b0aec5ad8ae-kube-api-access-cd6rs\") pod \"cert-manager-cainjector-559cdd8c87-frwx2\" (UID: \"35444ca7-f346-4d21-ba2d-6b0aec5ad8ae\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-frwx2"
Sep 10 15:46:59.839354 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.839318    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd6rs\" (UniqueName: \"kubernetes.io/projected/35444ca7-f346-4d21-ba2d-6b0aec5ad8ae-kube-api-access-cd6rs\") pod \"cert-manager-cainjector-559cdd8c87-frwx2\" (UID: \"35444ca7-f346-4d21-ba2d-6b0aec5ad8ae\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-frwx2"
Sep 10 15:46:59.839537 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.839391    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35444ca7-f346-4d21-ba2d-6b0aec5ad8ae-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-frwx2\" (UID: \"35444ca7-f346-4d21-ba2d-6b0aec5ad8ae\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-frwx2"
Sep 10 15:46:59.847924 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.847885    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd6rs\" (UniqueName: \"kubernetes.io/projected/35444ca7-f346-4d21-ba2d-6b0aec5ad8ae-kube-api-access-cd6rs\") pod \"cert-manager-cainjector-559cdd8c87-frwx2\" (UID: \"35444ca7-f346-4d21-ba2d-6b0aec5ad8ae\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-frwx2"
Sep 10 15:46:59.847924 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.847898    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35444ca7-f346-4d21-ba2d-6b0aec5ad8ae-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-frwx2\" (UID: \"35444ca7-f346-4d21-ba2d-6b0aec5ad8ae\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-frwx2"
Sep 10 15:46:59.980120 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:46:59.980078    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-frwx2"
Sep 10 15:47:00.194450 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:00.194417    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd" event={"ID":"b4940696-5525-43f0-b39b-7d3cdd5518a6","Type":"ContainerDied","Data":"12508df7d300cfab40dafadc72e97144b45b2d433e44419046dac80d93239569"}
Sep 10 15:47:00.194590 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:00.194456    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7dmbd"
Sep 10 15:47:00.194590 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:00.194464    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12508df7d300cfab40dafadc72e97144b45b2d433e44419046dac80d93239569"
Sep 10 15:47:00.252485 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:00.252291    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-frwx2"]
Sep 10 15:47:00.254804 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:47:00.254780    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35444ca7_f346_4d21_ba2d_6b0aec5ad8ae.slice/crio-e9489a15055f571e216b1d98563979b1ab918b37df501ab1835749b50bc13cfe WatchSource:0}: Error finding container e9489a15055f571e216b1d98563979b1ab918b37df501ab1835749b50bc13cfe: Status 404 returned error can't find the container with id e9489a15055f571e216b1d98563979b1ab918b37df501ab1835749b50bc13cfe
Sep 10 15:47:01.201958 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:01.201922    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-frwx2" event={"ID":"35444ca7-f346-4d21-ba2d-6b0aec5ad8ae","Type":"ContainerStarted","Data":"e9489a15055f571e216b1d98563979b1ab918b37df501ab1835749b50bc13cfe"}
Sep 10 15:47:02.683560 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:02.683523    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-p74kl"]
Sep 10 15:47:02.687351 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:02.687326    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-p74kl"
Sep 10 15:47:02.689497 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:02.689474    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-dqz7v\""
Sep 10 15:47:02.695054 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:02.695010    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-p74kl"]
Sep 10 15:47:02.869166 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:02.869126    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdtdm\" (UniqueName: \"kubernetes.io/projected/5c02afb9-ae17-47cf-b96b-bc92d6bedb2d-kube-api-access-tdtdm\") pod \"cert-manager-55b4d954d5-p74kl\" (UID: \"5c02afb9-ae17-47cf-b96b-bc92d6bedb2d\") " pod="cert-manager/cert-manager-55b4d954d5-p74kl"
Sep 10 15:47:02.869166 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:02.869173    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c02afb9-ae17-47cf-b96b-bc92d6bedb2d-bound-sa-token\") pod \"cert-manager-55b4d954d5-p74kl\" (UID: \"5c02afb9-ae17-47cf-b96b-bc92d6bedb2d\") " pod="cert-manager/cert-manager-55b4d954d5-p74kl"
Sep 10 15:47:02.970050 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:02.969953    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdtdm\" (UniqueName: \"kubernetes.io/projected/5c02afb9-ae17-47cf-b96b-bc92d6bedb2d-kube-api-access-tdtdm\") pod \"cert-manager-55b4d954d5-p74kl\" (UID: \"5c02afb9-ae17-47cf-b96b-bc92d6bedb2d\") " pod="cert-manager/cert-manager-55b4d954d5-p74kl"
Sep 10 15:47:02.970050 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:02.970010    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c02afb9-ae17-47cf-b96b-bc92d6bedb2d-bound-sa-token\") pod \"cert-manager-55b4d954d5-p74kl\" (UID: \"5c02afb9-ae17-47cf-b96b-bc92d6bedb2d\") " pod="cert-manager/cert-manager-55b4d954d5-p74kl"
Sep 10 15:47:02.978257 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:02.978220    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c02afb9-ae17-47cf-b96b-bc92d6bedb2d-bound-sa-token\") pod \"cert-manager-55b4d954d5-p74kl\" (UID: \"5c02afb9-ae17-47cf-b96b-bc92d6bedb2d\") " pod="cert-manager/cert-manager-55b4d954d5-p74kl"
Sep 10 15:47:02.978474 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:02.978453    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdtdm\" (UniqueName: \"kubernetes.io/projected/5c02afb9-ae17-47cf-b96b-bc92d6bedb2d-kube-api-access-tdtdm\") pod \"cert-manager-55b4d954d5-p74kl\" (UID: \"5c02afb9-ae17-47cf-b96b-bc92d6bedb2d\") " pod="cert-manager/cert-manager-55b4d954d5-p74kl"
Sep 10 15:47:02.998683 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:02.998652    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-p74kl"
Sep 10 15:47:03.212750 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:03.212709    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-frwx2" event={"ID":"35444ca7-f346-4d21-ba2d-6b0aec5ad8ae","Type":"ContainerStarted","Data":"b5bd16cbf0bca79d4364920eb233787ef5ba3e827ce8c148e8311cfd0a9f3b97"}
Sep 10 15:47:03.237405 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:03.237284    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-frwx2" podStartSLOduration=1.725751252 podStartE2EDuration="4.23726768s" totalImagesPullingTime="2.511516428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:03.234750205 +0000 UTC m=+295.907160291" watchObservedRunningTime="2026-09-10 15:47:03.23726768 +0000 UTC m=+295.909677763"
Sep 10 15:47:03.277666 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:03.277629    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-p74kl"]
Sep 10 15:47:03.281446 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:47:03.281416    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c02afb9_ae17_47cf_b96b_bc92d6bedb2d.slice/crio-f800c8abc5e28070e28288d9d8102d4f185fed7ddb96c0fc1533ebced0f6d410 WatchSource:0}: Error finding container f800c8abc5e28070e28288d9d8102d4f185fed7ddb96c0fc1533ebced0f6d410: Status 404 returned error can't find the container with id f800c8abc5e28070e28288d9d8102d4f185fed7ddb96c0fc1533ebced0f6d410
Sep 10 15:47:04.218369 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:04.218326    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-p74kl" event={"ID":"5c02afb9-ae17-47cf-b96b-bc92d6bedb2d","Type":"ContainerStarted","Data":"de41ac581fb0010a67a8655736af24d12907ac2922215c15c2700ed87be61dd2"}
Sep 10 15:47:04.218841 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:04.218379    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-p74kl" event={"ID":"5c02afb9-ae17-47cf-b96b-bc92d6bedb2d","Type":"ContainerStarted","Data":"f800c8abc5e28070e28288d9d8102d4f185fed7ddb96c0fc1533ebced0f6d410"}
Sep 10 15:47:04.237717 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:04.237660    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-p74kl" podStartSLOduration=2.237642378 podStartE2EDuration="2.237642378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:04.236241097 +0000 UTC m=+296.908651179" watchObservedRunningTime="2026-09-10 15:47:04.237642378 +0000 UTC m=+296.910052460"
Sep 10 15:47:07.752320 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:07.752273    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7sm72_8ddd4dc1-1122-41e0-a54a-67fb8008d7b9/ovn-acl-logging/0.log"
Sep 10 15:47:07.753920 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:07.753895    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7sm72_8ddd4dc1-1122-41e0-a54a-67fb8008d7b9/ovn-acl-logging/0.log"
Sep 10 15:47:07.755867 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:07.755848    2572 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 15:47:12.799671 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:12.799636    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl"]
Sep 10 15:47:12.818439 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:12.818408    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl"]
Sep 10 15:47:12.818575 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:12.818559    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl"
Sep 10 15:47:12.822945 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:12.822917    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 15:47:12.823106 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:12.822996    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 15:47:12.823651 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:12.823636    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hg86r\""
Sep 10 15:47:12.857747 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:12.857710    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn9jr\" (UniqueName: \"kubernetes.io/projected/f0ddace0-6229-453d-b061-f04cf7bb2279-kube-api-access-jn9jr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl\" (UID: \"f0ddace0-6229-453d-b061-f04cf7bb2279\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl"
Sep 10 15:47:12.857747 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:12.857753    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0ddace0-6229-453d-b061-f04cf7bb2279-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl\" (UID: \"f0ddace0-6229-453d-b061-f04cf7bb2279\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl"
Sep 10 15:47:12.857974 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:12.857857    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0ddace0-6229-453d-b061-f04cf7bb2279-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl\" (UID: \"f0ddace0-6229-453d-b061-f04cf7bb2279\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl"
Sep 10 15:47:12.959117 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:12.959074    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0ddace0-6229-453d-b061-f04cf7bb2279-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl\" (UID: \"f0ddace0-6229-453d-b061-f04cf7bb2279\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl"
Sep 10 15:47:12.959290 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:12.959132    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn9jr\" (UniqueName: \"kubernetes.io/projected/f0ddace0-6229-453d-b061-f04cf7bb2279-kube-api-access-jn9jr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl\" (UID: \"f0ddace0-6229-453d-b061-f04cf7bb2279\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl"
Sep 10 15:47:12.959290 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:12.959167    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0ddace0-6229-453d-b061-f04cf7bb2279-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl\" (UID: \"f0ddace0-6229-453d-b061-f04cf7bb2279\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl"
Sep 10 15:47:12.959501 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:12.959480    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0ddace0-6229-453d-b061-f04cf7bb2279-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl\" (UID: \"f0ddace0-6229-453d-b061-f04cf7bb2279\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl"
Sep 10 15:47:12.959569 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:12.959548    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0ddace0-6229-453d-b061-f04cf7bb2279-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl\" (UID: \"f0ddace0-6229-453d-b061-f04cf7bb2279\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl"
Sep 10 15:47:12.968174 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:12.968150    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn9jr\" (UniqueName: \"kubernetes.io/projected/f0ddace0-6229-453d-b061-f04cf7bb2279-kube-api-access-jn9jr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl\" (UID: \"f0ddace0-6229-453d-b061-f04cf7bb2279\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl"
Sep 10 15:47:13.130766 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:13.130670    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl"
Sep 10 15:47:13.418769 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:13.418673    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl"]
Sep 10 15:47:13.421682 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:47:13.421650    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ddace0_6229_453d_b061_f04cf7bb2279.slice/crio-9aefa098b9fb258a94497032006d107c28db5b0eff560d312918d571d20dc75a WatchSource:0}: Error finding container 9aefa098b9fb258a94497032006d107c28db5b0eff560d312918d571d20dc75a: Status 404 returned error can't find the container with id 9aefa098b9fb258a94497032006d107c28db5b0eff560d312918d571d20dc75a
Sep 10 15:47:14.262948 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:14.262860    2572 generic.go:356] "Generic (PLEG): container finished" podID="f0ddace0-6229-453d-b061-f04cf7bb2279" containerID="6f2e7f289f05e7a94809674536c3ea729707f068ca7d9d30eeca3f0556e7dca2" exitCode=0
Sep 10 15:47:14.262948 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:14.262935    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl" event={"ID":"f0ddace0-6229-453d-b061-f04cf7bb2279","Type":"ContainerDied","Data":"6f2e7f289f05e7a94809674536c3ea729707f068ca7d9d30eeca3f0556e7dca2"}
Sep 10 15:47:14.263374 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:14.262977    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl" event={"ID":"f0ddace0-6229-453d-b061-f04cf7bb2279","Type":"ContainerStarted","Data":"9aefa098b9fb258a94497032006d107c28db5b0eff560d312918d571d20dc75a"}
Sep 10 15:47:14.263463 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:14.263448    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:47:15.269539 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:15.269497    2572 generic.go:356] "Generic (PLEG): container finished" podID="f0ddace0-6229-453d-b061-f04cf7bb2279" containerID="d08aa389424c05e47dfa8cd50922030ad153720fe3e507dd0382121dfe43c052" exitCode=0
Sep 10 15:47:15.269893 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:15.269580    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl" event={"ID":"f0ddace0-6229-453d-b061-f04cf7bb2279","Type":"ContainerDied","Data":"d08aa389424c05e47dfa8cd50922030ad153720fe3e507dd0382121dfe43c052"}
Sep 10 15:47:16.276319 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:16.276271    2572 generic.go:356] "Generic (PLEG): container finished" podID="f0ddace0-6229-453d-b061-f04cf7bb2279" containerID="ff8f60861a91d1257390859ec04dbbabb9054eb4cc76377eac00d21755efeb93" exitCode=0
Sep 10 15:47:16.276719 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:16.276354    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl" event={"ID":"f0ddace0-6229-453d-b061-f04cf7bb2279","Type":"ContainerDied","Data":"ff8f60861a91d1257390859ec04dbbabb9054eb4cc76377eac00d21755efeb93"}
Sep 10 15:47:17.402105 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:17.402082    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl"
Sep 10 15:47:17.494131 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:17.494092    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn9jr\" (UniqueName: \"kubernetes.io/projected/f0ddace0-6229-453d-b061-f04cf7bb2279-kube-api-access-jn9jr\") pod \"f0ddace0-6229-453d-b061-f04cf7bb2279\" (UID: \"f0ddace0-6229-453d-b061-f04cf7bb2279\") "
Sep 10 15:47:17.494364 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:17.494152    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0ddace0-6229-453d-b061-f04cf7bb2279-util\") pod \"f0ddace0-6229-453d-b061-f04cf7bb2279\" (UID: \"f0ddace0-6229-453d-b061-f04cf7bb2279\") "
Sep 10 15:47:17.494364 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:17.494184    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0ddace0-6229-453d-b061-f04cf7bb2279-bundle\") pod \"f0ddace0-6229-453d-b061-f04cf7bb2279\" (UID: \"f0ddace0-6229-453d-b061-f04cf7bb2279\") "
Sep 10 15:47:17.495105 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:17.495063    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ddace0-6229-453d-b061-f04cf7bb2279-bundle" (OuterVolumeSpecName: "bundle") pod "f0ddace0-6229-453d-b061-f04cf7bb2279" (UID: "f0ddace0-6229-453d-b061-f04cf7bb2279"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:47:17.496243 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:17.496216    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ddace0-6229-453d-b061-f04cf7bb2279-kube-api-access-jn9jr" (OuterVolumeSpecName: "kube-api-access-jn9jr") pod "f0ddace0-6229-453d-b061-f04cf7bb2279" (UID: "f0ddace0-6229-453d-b061-f04cf7bb2279"). InnerVolumeSpecName "kube-api-access-jn9jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:47:17.502723 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:17.502698    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ddace0-6229-453d-b061-f04cf7bb2279-util" (OuterVolumeSpecName: "util") pod "f0ddace0-6229-453d-b061-f04cf7bb2279" (UID: "f0ddace0-6229-453d-b061-f04cf7bb2279"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:47:17.595571 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:17.595540    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn9jr\" (UniqueName: \"kubernetes.io/projected/f0ddace0-6229-453d-b061-f04cf7bb2279-kube-api-access-jn9jr\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:47:17.595571 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:17.595569    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0ddace0-6229-453d-b061-f04cf7bb2279-util\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:47:17.595757 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:17.595579    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0ddace0-6229-453d-b061-f04cf7bb2279-bundle\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:47:18.286824 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:18.286733    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl" event={"ID":"f0ddace0-6229-453d-b061-f04cf7bb2279","Type":"ContainerDied","Data":"9aefa098b9fb258a94497032006d107c28db5b0eff560d312918d571d20dc75a"}
Sep 10 15:47:18.286824 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:18.286755    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8kjl"
Sep 10 15:47:18.286824 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:18.286787    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9aefa098b9fb258a94497032006d107c28db5b0eff560d312918d571d20dc75a"
Sep 10 15:47:29.901890 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.901852    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"]
Sep 10 15:47:29.902481 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.902207    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ddace0-6229-453d-b061-f04cf7bb2279" containerName="util"
Sep 10 15:47:29.902481 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.902221    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ddace0-6229-453d-b061-f04cf7bb2279" containerName="util"
Sep 10 15:47:29.902481 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.902237    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ddace0-6229-453d-b061-f04cf7bb2279" containerName="pull"
Sep 10 15:47:29.902481 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.902252    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ddace0-6229-453d-b061-f04cf7bb2279" containerName="pull"
Sep 10 15:47:29.902481 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.902267    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ddace0-6229-453d-b061-f04cf7bb2279" containerName="extract"
Sep 10 15:47:29.902481 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.902273    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ddace0-6229-453d-b061-f04cf7bb2279" containerName="extract"
Sep 10 15:47:29.902481 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.902341    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0ddace0-6229-453d-b061-f04cf7bb2279" containerName="extract"
Sep 10 15:47:29.908068 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.908050    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"
Sep 10 15:47:29.912730 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.912710    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 15:47:29.913680 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.913654    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 15:47:29.914687 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.914669    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wwh6s\""
Sep 10 15:47:29.914830 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.914795    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 15:47:29.917893 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.917874    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 15:47:29.943763 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.943725    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"]
Sep 10 15:47:29.990462 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.990423    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj4hk\" (UniqueName: \"kubernetes.io/projected/f6a6c38a-156a-4e8b-8870-2fdf8a08a433-kube-api-access-lj4hk\") pod \"opendatahub-operator-controller-manager-97f7948ff-nthf5\" (UID: \"f6a6c38a-156a-4e8b-8870-2fdf8a08a433\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"
Sep 10 15:47:29.990668 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.990482    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f6a6c38a-156a-4e8b-8870-2fdf8a08a433-apiservice-cert\") pod \"opendatahub-operator-controller-manager-97f7948ff-nthf5\" (UID: \"f6a6c38a-156a-4e8b-8870-2fdf8a08a433\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"
Sep 10 15:47:29.990668 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.990551    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f6a6c38a-156a-4e8b-8870-2fdf8a08a433-manifests\") pod \"opendatahub-operator-controller-manager-97f7948ff-nthf5\" (UID: \"f6a6c38a-156a-4e8b-8870-2fdf8a08a433\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"
Sep 10 15:47:29.990668 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.990593    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6a6c38a-156a-4e8b-8870-2fdf8a08a433-webhook-cert\") pod \"opendatahub-operator-controller-manager-97f7948ff-nthf5\" (UID: \"f6a6c38a-156a-4e8b-8870-2fdf8a08a433\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"
Sep 10 15:47:29.990668 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:29.990656    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6a6c38a-156a-4e8b-8870-2fdf8a08a433-tmp\") pod \"opendatahub-operator-controller-manager-97f7948ff-nthf5\" (UID: \"f6a6c38a-156a-4e8b-8870-2fdf8a08a433\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"
Sep 10 15:47:30.092004 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.091958    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6a6c38a-156a-4e8b-8870-2fdf8a08a433-webhook-cert\") pod \"opendatahub-operator-controller-manager-97f7948ff-nthf5\" (UID: \"f6a6c38a-156a-4e8b-8870-2fdf8a08a433\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"
Sep 10 15:47:30.092214 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.092033    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6a6c38a-156a-4e8b-8870-2fdf8a08a433-tmp\") pod \"opendatahub-operator-controller-manager-97f7948ff-nthf5\" (UID: \"f6a6c38a-156a-4e8b-8870-2fdf8a08a433\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"
Sep 10 15:47:30.092214 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.092109    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj4hk\" (UniqueName: \"kubernetes.io/projected/f6a6c38a-156a-4e8b-8870-2fdf8a08a433-kube-api-access-lj4hk\") pod \"opendatahub-operator-controller-manager-97f7948ff-nthf5\" (UID: \"f6a6c38a-156a-4e8b-8870-2fdf8a08a433\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"
Sep 10 15:47:30.092214 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.092155    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f6a6c38a-156a-4e8b-8870-2fdf8a08a433-apiservice-cert\") pod \"opendatahub-operator-controller-manager-97f7948ff-nthf5\" (UID: \"f6a6c38a-156a-4e8b-8870-2fdf8a08a433\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"
Sep 10 15:47:30.092214 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.092183    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f6a6c38a-156a-4e8b-8870-2fdf8a08a433-manifests\") pod \"opendatahub-operator-controller-manager-97f7948ff-nthf5\" (UID: \"f6a6c38a-156a-4e8b-8870-2fdf8a08a433\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"
Sep 10 15:47:30.092745 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.092718    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f6a6c38a-156a-4e8b-8870-2fdf8a08a433-manifests\") pod \"opendatahub-operator-controller-manager-97f7948ff-nthf5\" (UID: \"f6a6c38a-156a-4e8b-8870-2fdf8a08a433\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"
Sep 10 15:47:30.094482 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.094460    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6a6c38a-156a-4e8b-8870-2fdf8a08a433-tmp\") pod \"opendatahub-operator-controller-manager-97f7948ff-nthf5\" (UID: \"f6a6c38a-156a-4e8b-8870-2fdf8a08a433\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"
Sep 10 15:47:30.094703 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.094684    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6a6c38a-156a-4e8b-8870-2fdf8a08a433-webhook-cert\") pod \"opendatahub-operator-controller-manager-97f7948ff-nthf5\" (UID: \"f6a6c38a-156a-4e8b-8870-2fdf8a08a433\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"
Sep 10 15:47:30.094824 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.094804    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f6a6c38a-156a-4e8b-8870-2fdf8a08a433-apiservice-cert\") pod \"opendatahub-operator-controller-manager-97f7948ff-nthf5\" (UID: \"f6a6c38a-156a-4e8b-8870-2fdf8a08a433\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"
Sep 10 15:47:30.104412 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.104388    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj4hk\" (UniqueName: \"kubernetes.io/projected/f6a6c38a-156a-4e8b-8870-2fdf8a08a433-kube-api-access-lj4hk\") pod \"opendatahub-operator-controller-manager-97f7948ff-nthf5\" (UID: \"f6a6c38a-156a-4e8b-8870-2fdf8a08a433\") " pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"
Sep 10 15:47:30.218113 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.218017    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"
Sep 10 15:47:30.403081 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.403048    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr"]
Sep 10 15:47:30.408770 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.408743    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr"
Sep 10 15:47:30.411164 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.411127    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 15:47:30.411164 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.411155    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hg86r\""
Sep 10 15:47:30.411371 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.411138    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 15:47:30.416591 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.416563    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr"]
Sep 10 15:47:30.495722 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.495407    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"]
Sep 10 15:47:30.495722 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.495703    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a486b65-c3cb-4432-b7d8-0bd22d07b72e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr\" (UID: \"4a486b65-c3cb-4432-b7d8-0bd22d07b72e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr"
Sep 10 15:47:30.495848 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.495764    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a486b65-c3cb-4432-b7d8-0bd22d07b72e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr\" (UID: \"4a486b65-c3cb-4432-b7d8-0bd22d07b72e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr"
Sep 10 15:47:30.495848 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.495825    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54x7m\" (UniqueName: \"kubernetes.io/projected/4a486b65-c3cb-4432-b7d8-0bd22d07b72e-kube-api-access-54x7m\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr\" (UID: \"4a486b65-c3cb-4432-b7d8-0bd22d07b72e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr"
Sep 10 15:47:30.497947 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:47:30.497922    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a6c38a_156a_4e8b_8870_2fdf8a08a433.slice/crio-18646b265c734d30c58104d221b13a52b922d72473008323ef16a5f93ef780bd WatchSource:0}: Error finding container 18646b265c734d30c58104d221b13a52b922d72473008323ef16a5f93ef780bd: Status 404 returned error can't find the container with id 18646b265c734d30c58104d221b13a52b922d72473008323ef16a5f93ef780bd
Sep 10 15:47:30.596242 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.596200    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a486b65-c3cb-4432-b7d8-0bd22d07b72e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr\" (UID: \"4a486b65-c3cb-4432-b7d8-0bd22d07b72e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr"
Sep 10 15:47:30.596466 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.596252    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a486b65-c3cb-4432-b7d8-0bd22d07b72e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr\" (UID: \"4a486b65-c3cb-4432-b7d8-0bd22d07b72e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr"
Sep 10 15:47:30.596466 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.596279    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54x7m\" (UniqueName: \"kubernetes.io/projected/4a486b65-c3cb-4432-b7d8-0bd22d07b72e-kube-api-access-54x7m\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr\" (UID: \"4a486b65-c3cb-4432-b7d8-0bd22d07b72e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr"
Sep 10 15:47:30.596757 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.596731    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a486b65-c3cb-4432-b7d8-0bd22d07b72e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr\" (UID: \"4a486b65-c3cb-4432-b7d8-0bd22d07b72e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr"
Sep 10 15:47:30.596838 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.596743    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a486b65-c3cb-4432-b7d8-0bd22d07b72e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr\" (UID: \"4a486b65-c3cb-4432-b7d8-0bd22d07b72e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr"
Sep 10 15:47:30.609315 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.609266    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54x7m\" (UniqueName: \"kubernetes.io/projected/4a486b65-c3cb-4432-b7d8-0bd22d07b72e-kube-api-access-54x7m\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr\" (UID: \"4a486b65-c3cb-4432-b7d8-0bd22d07b72e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr"
Sep 10 15:47:30.719188 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:30.719144    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr"
Sep 10 15:47:31.000452 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:31.000424    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr"]
Sep 10 15:47:31.002970 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:47:31.002931    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a486b65_c3cb_4432_b7d8_0bd22d07b72e.slice/crio-23ab441dddf773c4cbbef21dac131b720dde12c0d6be15efd12c81b8e374e275 WatchSource:0}: Error finding container 23ab441dddf773c4cbbef21dac131b720dde12c0d6be15efd12c81b8e374e275: Status 404 returned error can't find the container with id 23ab441dddf773c4cbbef21dac131b720dde12c0d6be15efd12c81b8e374e275
Sep 10 15:47:31.345772 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:31.345733    2572 generic.go:356] "Generic (PLEG): container finished" podID="4a486b65-c3cb-4432-b7d8-0bd22d07b72e" containerID="e689256c0b9de5318168c90ad3711ffd50118f6f8be39da5ad9d52f8a89df7cf" exitCode=0
Sep 10 15:47:31.345951 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:31.345818    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr" event={"ID":"4a486b65-c3cb-4432-b7d8-0bd22d07b72e","Type":"ContainerDied","Data":"e689256c0b9de5318168c90ad3711ffd50118f6f8be39da5ad9d52f8a89df7cf"}
Sep 10 15:47:31.345951 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:31.345875    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr" event={"ID":"4a486b65-c3cb-4432-b7d8-0bd22d07b72e","Type":"ContainerStarted","Data":"23ab441dddf773c4cbbef21dac131b720dde12c0d6be15efd12c81b8e374e275"}
Sep 10 15:47:31.347351 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:31.347330    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5" event={"ID":"f6a6c38a-156a-4e8b-8870-2fdf8a08a433","Type":"ContainerStarted","Data":"18646b265c734d30c58104d221b13a52b922d72473008323ef16a5f93ef780bd"}
Sep 10 15:47:34.365661 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:34.365570    2572 generic.go:356] "Generic (PLEG): container finished" podID="4a486b65-c3cb-4432-b7d8-0bd22d07b72e" containerID="1a1835c573b056169879ebad3d8e7d37e0b21373f77717f315b1e1bd1379498a" exitCode=0
Sep 10 15:47:34.366048 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:34.365662    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr" event={"ID":"4a486b65-c3cb-4432-b7d8-0bd22d07b72e","Type":"ContainerDied","Data":"1a1835c573b056169879ebad3d8e7d37e0b21373f77717f315b1e1bd1379498a"}
Sep 10 15:47:34.367445 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:34.367417    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5" event={"ID":"f6a6c38a-156a-4e8b-8870-2fdf8a08a433","Type":"ContainerStarted","Data":"a09ea67ddf2f9fa39377b34a42f65211d7b9dadd9c229c89ab27530f644522ae"}
Sep 10 15:47:35.374140 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:35.374105    2572 generic.go:356] "Generic (PLEG): container finished" podID="4a486b65-c3cb-4432-b7d8-0bd22d07b72e" containerID="8f528cdaa99092dbcca41e274743438ddbb25438009e4f62a5c5258ea9d443ed" exitCode=0
Sep 10 15:47:35.374645 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:35.374187    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr" event={"ID":"4a486b65-c3cb-4432-b7d8-0bd22d07b72e","Type":"ContainerDied","Data":"8f528cdaa99092dbcca41e274743438ddbb25438009e4f62a5c5258ea9d443ed"}
Sep 10 15:47:35.375995 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:35.375968    2572 generic.go:356] "Generic (PLEG): container finished" podID="f6a6c38a-156a-4e8b-8870-2fdf8a08a433" containerID="a09ea67ddf2f9fa39377b34a42f65211d7b9dadd9c229c89ab27530f644522ae" exitCode=0
Sep 10 15:47:35.376118 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:35.376057    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5" event={"ID":"f6a6c38a-156a-4e8b-8870-2fdf8a08a433","Type":"ContainerDied","Data":"a09ea67ddf2f9fa39377b34a42f65211d7b9dadd9c229c89ab27530f644522ae"}
Sep 10 15:47:36.530634 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:36.530599    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr"
Sep 10 15:47:36.652051 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:36.651947    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a486b65-c3cb-4432-b7d8-0bd22d07b72e-bundle\") pod \"4a486b65-c3cb-4432-b7d8-0bd22d07b72e\" (UID: \"4a486b65-c3cb-4432-b7d8-0bd22d07b72e\") "
Sep 10 15:47:36.652051 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:36.652014    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a486b65-c3cb-4432-b7d8-0bd22d07b72e-util\") pod \"4a486b65-c3cb-4432-b7d8-0bd22d07b72e\" (UID: \"4a486b65-c3cb-4432-b7d8-0bd22d07b72e\") "
Sep 10 15:47:36.652296 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:36.652140    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54x7m\" (UniqueName: \"kubernetes.io/projected/4a486b65-c3cb-4432-b7d8-0bd22d07b72e-kube-api-access-54x7m\") pod \"4a486b65-c3cb-4432-b7d8-0bd22d07b72e\" (UID: \"4a486b65-c3cb-4432-b7d8-0bd22d07b72e\") "
Sep 10 15:47:36.652674 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:36.652648    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a486b65-c3cb-4432-b7d8-0bd22d07b72e-bundle" (OuterVolumeSpecName: "bundle") pod "4a486b65-c3cb-4432-b7d8-0bd22d07b72e" (UID: "4a486b65-c3cb-4432-b7d8-0bd22d07b72e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:47:36.654334 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:36.654294    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a486b65-c3cb-4432-b7d8-0bd22d07b72e-kube-api-access-54x7m" (OuterVolumeSpecName: "kube-api-access-54x7m") pod "4a486b65-c3cb-4432-b7d8-0bd22d07b72e" (UID: "4a486b65-c3cb-4432-b7d8-0bd22d07b72e"). InnerVolumeSpecName "kube-api-access-54x7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:47:36.657434 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:36.657393    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a486b65-c3cb-4432-b7d8-0bd22d07b72e-util" (OuterVolumeSpecName: "util") pod "4a486b65-c3cb-4432-b7d8-0bd22d07b72e" (UID: "4a486b65-c3cb-4432-b7d8-0bd22d07b72e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:47:36.753061 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:36.753014    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54x7m\" (UniqueName: \"kubernetes.io/projected/4a486b65-c3cb-4432-b7d8-0bd22d07b72e-kube-api-access-54x7m\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:47:36.753061 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:36.753055    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a486b65-c3cb-4432-b7d8-0bd22d07b72e-bundle\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:47:36.753061 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:36.753067    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a486b65-c3cb-4432-b7d8-0bd22d07b72e-util\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:47:37.386516 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:37.386479    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr" event={"ID":"4a486b65-c3cb-4432-b7d8-0bd22d07b72e","Type":"ContainerDied","Data":"23ab441dddf773c4cbbef21dac131b720dde12c0d6be15efd12c81b8e374e275"}
Sep 10 15:47:37.386516 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:37.386495    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0d67pr"
Sep 10 15:47:37.386743 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:37.386528    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23ab441dddf773c4cbbef21dac131b720dde12c0d6be15efd12c81b8e374e275"
Sep 10 15:47:37.388751 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:37.388719    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5" event={"ID":"f6a6c38a-156a-4e8b-8870-2fdf8a08a433","Type":"ContainerStarted","Data":"e0dca80790343a091c3b809867cc00f6094a9bf34cd4d09c972d68096bcf0a87"}
Sep 10 15:47:37.388914 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:37.388820    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"
Sep 10 15:47:37.409363 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:37.409269    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5" podStartSLOduration=2.78275515 podStartE2EDuration="8.409249177s" totalImagesPullingTime="4.619216906s" totalInitContainerRuntime="1.007277121s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:37.407355565 +0000 UTC m=+330.079765664" watchObservedRunningTime="2026-09-10 15:47:37.409249177 +0000 UTC m=+330.081659261"
Sep 10 15:47:48.395739 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:48.395706    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-97f7948ff-nthf5"
Sep 10 15:47:50.695655 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.695614    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-644bb569f8-4sghh"]
Sep 10 15:47:50.696066 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.695916    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a486b65-c3cb-4432-b7d8-0bd22d07b72e" containerName="pull"
Sep 10 15:47:50.696066 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.695927    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a486b65-c3cb-4432-b7d8-0bd22d07b72e" containerName="pull"
Sep 10 15:47:50.696066 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.695935    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a486b65-c3cb-4432-b7d8-0bd22d07b72e" containerName="extract"
Sep 10 15:47:50.696066 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.695941    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a486b65-c3cb-4432-b7d8-0bd22d07b72e" containerName="extract"
Sep 10 15:47:50.696066 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.695955    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a486b65-c3cb-4432-b7d8-0bd22d07b72e" containerName="util"
Sep 10 15:47:50.696066 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.695960    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a486b65-c3cb-4432-b7d8-0bd22d07b72e" containerName="util"
Sep 10 15:47:50.696066 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.696019    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a486b65-c3cb-4432-b7d8-0bd22d07b72e" containerName="extract"
Sep 10 15:47:50.700946 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.700926    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-644bb569f8-4sghh"
Sep 10 15:47:50.704473 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.704445    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7pq96\""
Sep 10 15:47:50.706358 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.706336    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 15:47:50.706529 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.706456    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 15:47:50.706529 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.706499    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 15:47:50.708334 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.708317    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 15:47:50.742136 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.742101    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-644bb569f8-4sghh"]
Sep 10 15:47:50.767357 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.767321    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8qv2\" (UniqueName: \"kubernetes.io/projected/8cc11c10-55e2-4a87-ab31-64835557d956-kube-api-access-z8qv2\") pod \"kube-auth-proxy-644bb569f8-4sghh\" (UID: \"8cc11c10-55e2-4a87-ab31-64835557d956\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-4sghh"
Sep 10 15:47:50.767537 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.767425    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cc11c10-55e2-4a87-ab31-64835557d956-tmp\") pod \"kube-auth-proxy-644bb569f8-4sghh\" (UID: \"8cc11c10-55e2-4a87-ab31-64835557d956\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-4sghh"
Sep 10 15:47:50.767537 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.767477    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc11c10-55e2-4a87-ab31-64835557d956-tls-certs\") pod \"kube-auth-proxy-644bb569f8-4sghh\" (UID: \"8cc11c10-55e2-4a87-ab31-64835557d956\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-4sghh"
Sep 10 15:47:50.868651 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.868611    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8qv2\" (UniqueName: \"kubernetes.io/projected/8cc11c10-55e2-4a87-ab31-64835557d956-kube-api-access-z8qv2\") pod \"kube-auth-proxy-644bb569f8-4sghh\" (UID: \"8cc11c10-55e2-4a87-ab31-64835557d956\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-4sghh"
Sep 10 15:47:50.868837 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.868665    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cc11c10-55e2-4a87-ab31-64835557d956-tmp\") pod \"kube-auth-proxy-644bb569f8-4sghh\" (UID: \"8cc11c10-55e2-4a87-ab31-64835557d956\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-4sghh"
Sep 10 15:47:50.868837 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.868692    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc11c10-55e2-4a87-ab31-64835557d956-tls-certs\") pod \"kube-auth-proxy-644bb569f8-4sghh\" (UID: \"8cc11c10-55e2-4a87-ab31-64835557d956\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-4sghh"
Sep 10 15:47:50.868837 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:47:50.868828    2572 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 10 15:47:50.868941 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:47:50.868901    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc11c10-55e2-4a87-ab31-64835557d956-tls-certs podName:8cc11c10-55e2-4a87-ab31-64835557d956 nodeName:}" failed. No retries permitted until 2026-09-10 15:47:51.368882568 +0000 UTC m=+344.041292647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8cc11c10-55e2-4a87-ab31-64835557d956-tls-certs") pod "kube-auth-proxy-644bb569f8-4sghh" (UID: "8cc11c10-55e2-4a87-ab31-64835557d956") : secret "kube-auth-proxy-tls" not found
Sep 10 15:47:50.871041 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.871005    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cc11c10-55e2-4a87-ab31-64835557d956-tmp\") pod \"kube-auth-proxy-644bb569f8-4sghh\" (UID: \"8cc11c10-55e2-4a87-ab31-64835557d956\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-4sghh"
Sep 10 15:47:50.879992 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:50.879965    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8qv2\" (UniqueName: \"kubernetes.io/projected/8cc11c10-55e2-4a87-ab31-64835557d956-kube-api-access-z8qv2\") pod \"kube-auth-proxy-644bb569f8-4sghh\" (UID: \"8cc11c10-55e2-4a87-ab31-64835557d956\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-4sghh"
Sep 10 15:47:51.373586 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:51.373544    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc11c10-55e2-4a87-ab31-64835557d956-tls-certs\") pod \"kube-auth-proxy-644bb569f8-4sghh\" (UID: \"8cc11c10-55e2-4a87-ab31-64835557d956\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-4sghh"
Sep 10 15:47:51.376197 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:51.376163    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc11c10-55e2-4a87-ab31-64835557d956-tls-certs\") pod \"kube-auth-proxy-644bb569f8-4sghh\" (UID: \"8cc11c10-55e2-4a87-ab31-64835557d956\") " pod="openshift-ingress/kube-auth-proxy-644bb569f8-4sghh"
Sep 10 15:47:51.611178 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:51.611140    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-644bb569f8-4sghh"
Sep 10 15:47:51.895186 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:51.895144    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-644bb569f8-4sghh"]
Sep 10 15:47:51.898428 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:47:51.898392    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc11c10_55e2_4a87_ab31_64835557d956.slice/crio-a02b8403775e6fe454680779bcea532c2253301ccae2ba436e2f1a21680f351b WatchSource:0}: Error finding container a02b8403775e6fe454680779bcea532c2253301ccae2ba436e2f1a21680f351b: Status 404 returned error can't find the container with id a02b8403775e6fe454680779bcea532c2253301ccae2ba436e2f1a21680f351b
Sep 10 15:47:52.318817 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.318777    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"]
Sep 10 15:47:52.323271 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.323250    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.326251 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.326215    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 10 15:47:52.326549 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.326528    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 15:47:52.326757 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.326740    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 10 15:47:52.327040 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.327024    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 10 15:47:52.327530 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.327515    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-n7xm2\""
Sep 10 15:47:52.348686 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.348655    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"]
Sep 10 15:47:52.455563 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.455525    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-644bb569f8-4sghh" event={"ID":"8cc11c10-55e2-4a87-ab31-64835557d956","Type":"ContainerStarted","Data":"a02b8403775e6fe454680779bcea532c2253301ccae2ba436e2f1a21680f351b"}
Sep 10 15:47:52.482858 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.482822    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.482858 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.482859    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.483077 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.482884    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.483077 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.482976    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gff8d\" (UniqueName: \"kubernetes.io/projected/5e16212c-a33c-4976-b40a-95e3ca4aebd5-kube-api-access-gff8d\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.483077 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.483056    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.483169 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.483102    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e16212c-a33c-4976-b40a-95e3ca4aebd5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.483169 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.483126    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e16212c-a33c-4976-b40a-95e3ca4aebd5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.584690 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.584595    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e16212c-a33c-4976-b40a-95e3ca4aebd5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.584690 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.584648    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e16212c-a33c-4976-b40a-95e3ca4aebd5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.584923 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.584746    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.584923 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.584775    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.584923 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.584805    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.584923 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.584865    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gff8d\" (UniqueName: \"kubernetes.io/projected/5e16212c-a33c-4976-b40a-95e3ca4aebd5-kube-api-access-gff8d\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.584923 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.584915    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.585764 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.585691    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.587653 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.587617    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e16212c-a33c-4976-b40a-95e3ca4aebd5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.588205 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.588161    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e16212c-a33c-4976-b40a-95e3ca4aebd5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.588433 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.588408    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.589052 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.589012    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.595919 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.595891    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.596411 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.596387    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gff8d\" (UniqueName: \"kubernetes.io/projected/5e16212c-a33c-4976-b40a-95e3ca4aebd5-kube-api-access-gff8d\") pod \"istiod-openshift-gateway-7b7995d5cd-2wdvq\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.632888 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.632839    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:52.938120 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:52.936023    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"]
Sep 10 15:47:52.940029 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:47:52.939980    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e16212c_a33c_4976_b40a_95e3ca4aebd5.slice/crio-e7facc9f66451ffb6957ea3219ea47202e89f1ab3c5d868a6602fec4bf79881a WatchSource:0}: Error finding container e7facc9f66451ffb6957ea3219ea47202e89f1ab3c5d868a6602fec4bf79881a: Status 404 returned error can't find the container with id e7facc9f66451ffb6957ea3219ea47202e89f1ab3c5d868a6602fec4bf79881a
Sep 10 15:47:53.463997 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:53.463952    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq" event={"ID":"5e16212c-a33c-4976-b40a-95e3ca4aebd5","Type":"ContainerStarted","Data":"e7facc9f66451ffb6957ea3219ea47202e89f1ab3c5d868a6602fec4bf79881a"}
Sep 10 15:47:55.478115 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:55.478072    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-644bb569f8-4sghh" event={"ID":"8cc11c10-55e2-4a87-ab31-64835557d956","Type":"ContainerStarted","Data":"1b1de69333798f61eec1c4e33807526e3d52723ade320067124808473c4677f7"}
Sep 10 15:47:55.499103 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:55.499028    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-644bb569f8-4sghh" podStartSLOduration=2.7320221670000002 podStartE2EDuration="5.499003662s" totalImagesPullingTime="2.766981495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:55.498127402 +0000 UTC m=+348.170537496" watchObservedRunningTime="2026-09-10 15:47:55.499003662 +0000 UTC m=+348.171413749"
Sep 10 15:47:56.460353 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:56.460290    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:47:56.460444 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:56.460402    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:47:57.488556 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:57.488520    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq" event={"ID":"5e16212c-a33c-4976-b40a-95e3ca4aebd5","Type":"ContainerStarted","Data":"31919ff9eede860ff6905cb743976abc328891ab8f3a0b4f1179822f49c3cc42"}
Sep 10 15:47:57.489012 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:57.488689    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:57.490401 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:57.490379    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:47:57.507657 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:47:57.507605    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq" podStartSLOduration=1.990632097 podStartE2EDuration="5.507589457s" totalImagesPullingTime="3.51695736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:57.507272711 +0000 UTC m=+350.179682796" watchObservedRunningTime="2026-09-10 15:47:57.507589457 +0000 UTC m=+350.179999539"
Sep 10 15:48:08.562511 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.562471    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5"]
Sep 10 15:48:08.590192 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.590157    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5"
Sep 10 15:48:08.591928 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.591900    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5"]
Sep 10 15:48:08.593196 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.593175    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 15:48:08.593831 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.593806    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 15:48:08.594241 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.594218    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 15:48:08.594475 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.594453    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tbqck\""
Sep 10 15:48:08.594475 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.594468    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:48:08.594734 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.594497    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 15:48:08.732452 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.732411    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/787c6c8d-daa0-4bb3-967a-98fb1cc42d6d-cert\") pod \"lws-controller-manager-79dbb968b5-45wr5\" (UID: \"787c6c8d-daa0-4bb3-967a-98fb1cc42d6d\") " pod="openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5"
Sep 10 15:48:08.732630 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.732467    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/787c6c8d-daa0-4bb3-967a-98fb1cc42d6d-manager-config\") pod \"lws-controller-manager-79dbb968b5-45wr5\" (UID: \"787c6c8d-daa0-4bb3-967a-98fb1cc42d6d\") " pod="openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5"
Sep 10 15:48:08.732630 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.732487    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxb9f\" (UniqueName: \"kubernetes.io/projected/787c6c8d-daa0-4bb3-967a-98fb1cc42d6d-kube-api-access-kxb9f\") pod \"lws-controller-manager-79dbb968b5-45wr5\" (UID: \"787c6c8d-daa0-4bb3-967a-98fb1cc42d6d\") " pod="openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5"
Sep 10 15:48:08.732630 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.732565    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/787c6c8d-daa0-4bb3-967a-98fb1cc42d6d-metrics-cert\") pod \"lws-controller-manager-79dbb968b5-45wr5\" (UID: \"787c6c8d-daa0-4bb3-967a-98fb1cc42d6d\") " pod="openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5"
Sep 10 15:48:08.833222 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.833141    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/787c6c8d-daa0-4bb3-967a-98fb1cc42d6d-metrics-cert\") pod \"lws-controller-manager-79dbb968b5-45wr5\" (UID: \"787c6c8d-daa0-4bb3-967a-98fb1cc42d6d\") " pod="openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5"
Sep 10 15:48:08.833423 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.833242    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/787c6c8d-daa0-4bb3-967a-98fb1cc42d6d-cert\") pod \"lws-controller-manager-79dbb968b5-45wr5\" (UID: \"787c6c8d-daa0-4bb3-967a-98fb1cc42d6d\") " pod="openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5"
Sep 10 15:48:08.833423 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.833294    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/787c6c8d-daa0-4bb3-967a-98fb1cc42d6d-manager-config\") pod \"lws-controller-manager-79dbb968b5-45wr5\" (UID: \"787c6c8d-daa0-4bb3-967a-98fb1cc42d6d\") " pod="openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5"
Sep 10 15:48:08.833423 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.833331    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxb9f\" (UniqueName: \"kubernetes.io/projected/787c6c8d-daa0-4bb3-967a-98fb1cc42d6d-kube-api-access-kxb9f\") pod \"lws-controller-manager-79dbb968b5-45wr5\" (UID: \"787c6c8d-daa0-4bb3-967a-98fb1cc42d6d\") " pod="openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5"
Sep 10 15:48:08.834054 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.834030    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/787c6c8d-daa0-4bb3-967a-98fb1cc42d6d-manager-config\") pod \"lws-controller-manager-79dbb968b5-45wr5\" (UID: \"787c6c8d-daa0-4bb3-967a-98fb1cc42d6d\") " pod="openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5"
Sep 10 15:48:08.835756 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.835712    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/787c6c8d-daa0-4bb3-967a-98fb1cc42d6d-cert\") pod \"lws-controller-manager-79dbb968b5-45wr5\" (UID: \"787c6c8d-daa0-4bb3-967a-98fb1cc42d6d\") " pod="openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5"
Sep 10 15:48:08.835756 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.835753    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/787c6c8d-daa0-4bb3-967a-98fb1cc42d6d-metrics-cert\") pod \"lws-controller-manager-79dbb968b5-45wr5\" (UID: \"787c6c8d-daa0-4bb3-967a-98fb1cc42d6d\") " pod="openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5"
Sep 10 15:48:08.841950 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.841930    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxb9f\" (UniqueName: \"kubernetes.io/projected/787c6c8d-daa0-4bb3-967a-98fb1cc42d6d-kube-api-access-kxb9f\") pod \"lws-controller-manager-79dbb968b5-45wr5\" (UID: \"787c6c8d-daa0-4bb3-967a-98fb1cc42d6d\") " pod="openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5"
Sep 10 15:48:08.901494 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:08.901456    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5"
Sep 10 15:48:09.189158 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:09.189129    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5"]
Sep 10 15:48:09.191016 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:48:09.190983    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787c6c8d_daa0_4bb3_967a_98fb1cc42d6d.slice/crio-0e4bdb0a2e0e4f4a88fb8bdc24d4e667ee16c6d79c588b34ea34adc4c77d64cf WatchSource:0}: Error finding container 0e4bdb0a2e0e4f4a88fb8bdc24d4e667ee16c6d79c588b34ea34adc4c77d64cf: Status 404 returned error can't find the container with id 0e4bdb0a2e0e4f4a88fb8bdc24d4e667ee16c6d79c588b34ea34adc4c77d64cf
Sep 10 15:48:09.547648 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:09.547556    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5" event={"ID":"787c6c8d-daa0-4bb3-967a-98fb1cc42d6d","Type":"ContainerStarted","Data":"0e4bdb0a2e0e4f4a88fb8bdc24d4e667ee16c6d79c588b34ea34adc4c77d64cf"}
Sep 10 15:48:10.554802 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:10.554758    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5" event={"ID":"787c6c8d-daa0-4bb3-967a-98fb1cc42d6d","Type":"ContainerStarted","Data":"1594604dc6c8d8a1a0f01dea23638db7592392e528154a278483d0721519766f"}
Sep 10 15:48:10.555220 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:10.554978    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5"
Sep 10 15:48:10.619669 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:10.619609    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5" podStartSLOduration=1.438389884 podStartE2EDuration="2.619588658s" totalImagesPullingTime="1.181198774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:48:10.610893884 +0000 UTC m=+363.283303968" watchObservedRunningTime="2026-09-10 15:48:10.619588658 +0000 UTC m=+363.291998740"
Sep 10 15:48:12.930725 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:12.930686    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"]
Sep 10 15:48:12.931220 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:12.930898    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq" podUID="5e16212c-a33c-4976-b40a-95e3ca4aebd5" containerName="discovery" containerID="cri-o://31919ff9eede860ff6905cb743976abc328891ab8f3a0b4f1179822f49c3cc42" gracePeriod=30
Sep 10 15:48:13.166607 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.166582    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:48:13.264613 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.264517    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-kubeconfig\") pod \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") "
Sep 10 15:48:13.264613 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.264555    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gff8d\" (UniqueName: \"kubernetes.io/projected/5e16212c-a33c-4976-b40a-95e3ca4aebd5-kube-api-access-gff8d\") pod \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") "
Sep 10 15:48:13.264613 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.264577    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-csr-dns-cert\") pod \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") "
Sep 10 15:48:13.264613 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.264601    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-token\") pod \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") "
Sep 10 15:48:13.264925 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.264668    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e16212c-a33c-4976-b40a-95e3ca4aebd5-local-certs\") pod \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") "
Sep 10 15:48:13.264925 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.264692    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-csr-ca-configmap\") pod \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") "
Sep 10 15:48:13.264925 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.264760    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e16212c-a33c-4976-b40a-95e3ca4aebd5-cacerts\") pod \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\" (UID: \"5e16212c-a33c-4976-b40a-95e3ca4aebd5\") "
Sep 10 15:48:13.265132 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.265101    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "5e16212c-a33c-4976-b40a-95e3ca4aebd5" (UID: "5e16212c-a33c-4976-b40a-95e3ca4aebd5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:48:13.267069 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.267031    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e16212c-a33c-4976-b40a-95e3ca4aebd5-cacerts" (OuterVolumeSpecName: "cacerts") pod "5e16212c-a33c-4976-b40a-95e3ca4aebd5" (UID: "5e16212c-a33c-4976-b40a-95e3ca4aebd5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:48:13.267183 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.267113    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e16212c-a33c-4976-b40a-95e3ca4aebd5-local-certs" (OuterVolumeSpecName: "local-certs") pod "5e16212c-a33c-4976-b40a-95e3ca4aebd5" (UID: "5e16212c-a33c-4976-b40a-95e3ca4aebd5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:48:13.267231 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.267191    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "5e16212c-a33c-4976-b40a-95e3ca4aebd5" (UID: "5e16212c-a33c-4976-b40a-95e3ca4aebd5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:48:13.267231 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.267218    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e16212c-a33c-4976-b40a-95e3ca4aebd5-kube-api-access-gff8d" (OuterVolumeSpecName: "kube-api-access-gff8d") pod "5e16212c-a33c-4976-b40a-95e3ca4aebd5" (UID: "5e16212c-a33c-4976-b40a-95e3ca4aebd5"). InnerVolumeSpecName "kube-api-access-gff8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:48:13.267374 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.267334    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "5e16212c-a33c-4976-b40a-95e3ca4aebd5" (UID: "5e16212c-a33c-4976-b40a-95e3ca4aebd5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:48:13.267443 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.267427    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e16212c-a33c-4976-b40a-95e3ca4aebd5" (UID: "5e16212c-a33c-4976-b40a-95e3ca4aebd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:48:13.365316 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.365275    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-token\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:48:13.365489 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.365328    2572 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e16212c-a33c-4976-b40a-95e3ca4aebd5-local-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:48:13.365489 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.365340    2572 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-csr-ca-configmap\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:48:13.365489 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.365349    2572 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e16212c-a33c-4976-b40a-95e3ca4aebd5-cacerts\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:48:13.365489 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.365357    2572 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-kubeconfig\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:48:13.365489 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.365366    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gff8d\" (UniqueName: \"kubernetes.io/projected/5e16212c-a33c-4976-b40a-95e3ca4aebd5-kube-api-access-gff8d\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:48:13.365489 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.365375    2572 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e16212c-a33c-4976-b40a-95e3ca4aebd5-istio-csr-dns-cert\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:48:13.572390 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.572353    2572 generic.go:356] "Generic (PLEG): container finished" podID="5e16212c-a33c-4976-b40a-95e3ca4aebd5" containerID="31919ff9eede860ff6905cb743976abc328891ab8f3a0b4f1179822f49c3cc42" exitCode=0
Sep 10 15:48:13.572553 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.572416    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"
Sep 10 15:48:13.572553 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.572433    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq" event={"ID":"5e16212c-a33c-4976-b40a-95e3ca4aebd5","Type":"ContainerDied","Data":"31919ff9eede860ff6905cb743976abc328891ab8f3a0b4f1179822f49c3cc42"}
Sep 10 15:48:13.572553 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.572512    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq" event={"ID":"5e16212c-a33c-4976-b40a-95e3ca4aebd5","Type":"ContainerDied","Data":"e7facc9f66451ffb6957ea3219ea47202e89f1ab3c5d868a6602fec4bf79881a"}
Sep 10 15:48:13.572553 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.572533    2572 scope.go:117] "RemoveContainer" containerID="31919ff9eede860ff6905cb743976abc328891ab8f3a0b4f1179822f49c3cc42"
Sep 10 15:48:13.582598 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.582576    2572 scope.go:117] "RemoveContainer" containerID="31919ff9eede860ff6905cb743976abc328891ab8f3a0b4f1179822f49c3cc42"
Sep 10 15:48:13.582981 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:48:13.582947    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31919ff9eede860ff6905cb743976abc328891ab8f3a0b4f1179822f49c3cc42\": container with ID starting with 31919ff9eede860ff6905cb743976abc328891ab8f3a0b4f1179822f49c3cc42 not found: ID does not exist" containerID="31919ff9eede860ff6905cb743976abc328891ab8f3a0b4f1179822f49c3cc42"
Sep 10 15:48:13.583107 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.582989    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31919ff9eede860ff6905cb743976abc328891ab8f3a0b4f1179822f49c3cc42"} err="failed to get container status \"31919ff9eede860ff6905cb743976abc328891ab8f3a0b4f1179822f49c3cc42\": rpc error: code = NotFound desc = could not find container \"31919ff9eede860ff6905cb743976abc328891ab8f3a0b4f1179822f49c3cc42\": container with ID starting with 31919ff9eede860ff6905cb743976abc328891ab8f3a0b4f1179822f49c3cc42 not found: ID does not exist"
Sep 10 15:48:13.601760 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.601722    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"]
Sep 10 15:48:13.609049 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.609021    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2wdvq"]
Sep 10 15:48:13.992148 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:13.992069    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e16212c-a33c-4976-b40a-95e3ca4aebd5" path="/var/lib/kubelet/pods/5e16212c-a33c-4976-b40a-95e3ca4aebd5/volumes"
Sep 10 15:48:21.561484 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:48:21.561446    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-79dbb968b5-45wr5"
Sep 10 15:49:06.712826 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.712787    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65"]
Sep 10 15:49:06.713284 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.713102    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e16212c-a33c-4976-b40a-95e3ca4aebd5" containerName="discovery"
Sep 10 15:49:06.713284 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.713114    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e16212c-a33c-4976-b40a-95e3ca4aebd5" containerName="discovery"
Sep 10 15:49:06.713284 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.713166    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e16212c-a33c-4976-b40a-95e3ca4aebd5" containerName="discovery"
Sep 10 15:49:06.716384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.716367    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65"
Sep 10 15:49:06.719100 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.719070    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 15:49:06.719100 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.719090    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hg86r\""
Sep 10 15:49:06.719275 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.719070    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 15:49:06.723466 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.723441    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/469c7349-f352-4b2d-9db0-79cc0c05aa7a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65\" (UID: \"469c7349-f352-4b2d-9db0-79cc0c05aa7a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65"
Sep 10 15:49:06.723681 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.723494    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n66fj\" (UniqueName: \"kubernetes.io/projected/469c7349-f352-4b2d-9db0-79cc0c05aa7a-kube-api-access-n66fj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65\" (UID: \"469c7349-f352-4b2d-9db0-79cc0c05aa7a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65"
Sep 10 15:49:06.723681 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.723530    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/469c7349-f352-4b2d-9db0-79cc0c05aa7a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65\" (UID: \"469c7349-f352-4b2d-9db0-79cc0c05aa7a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65"
Sep 10 15:49:06.725610 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.725585    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65"]
Sep 10 15:49:06.807578 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.807534    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp"]
Sep 10 15:49:06.811135 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.811115    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp"
Sep 10 15:49:06.822166 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.822138    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp"]
Sep 10 15:49:06.824152 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.824122    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/469c7349-f352-4b2d-9db0-79cc0c05aa7a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65\" (UID: \"469c7349-f352-4b2d-9db0-79cc0c05aa7a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65"
Sep 10 15:49:06.824317 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.824163    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n66fj\" (UniqueName: \"kubernetes.io/projected/469c7349-f352-4b2d-9db0-79cc0c05aa7a-kube-api-access-n66fj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65\" (UID: \"469c7349-f352-4b2d-9db0-79cc0c05aa7a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65"
Sep 10 15:49:06.824317 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.824184    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/469c7349-f352-4b2d-9db0-79cc0c05aa7a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65\" (UID: \"469c7349-f352-4b2d-9db0-79cc0c05aa7a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65"
Sep 10 15:49:06.824618 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.824596    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/469c7349-f352-4b2d-9db0-79cc0c05aa7a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65\" (UID: \"469c7349-f352-4b2d-9db0-79cc0c05aa7a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65"
Sep 10 15:49:06.824691 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.824618    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/469c7349-f352-4b2d-9db0-79cc0c05aa7a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65\" (UID: \"469c7349-f352-4b2d-9db0-79cc0c05aa7a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65"
Sep 10 15:49:06.835082 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.835054    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n66fj\" (UniqueName: \"kubernetes.io/projected/469c7349-f352-4b2d-9db0-79cc0c05aa7a-kube-api-access-n66fj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65\" (UID: \"469c7349-f352-4b2d-9db0-79cc0c05aa7a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65"
Sep 10 15:49:06.908128 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.908090    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp"]
Sep 10 15:49:06.911964 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.911948    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp"
Sep 10 15:49:06.921599 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.921575    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp"]
Sep 10 15:49:06.924780 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.924755    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a536ffc-bdda-43ee-bb29-cb07eba27ff8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp\" (UID: \"8a536ffc-bdda-43ee-bb29-cb07eba27ff8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp"
Sep 10 15:49:06.924876 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.924808    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/411b311a-46f3-441a-b8f1-0473409b058b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp\" (UID: \"411b311a-46f3-441a-b8f1-0473409b058b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp"
Sep 10 15:49:06.924876 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.924842    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgfmf\" (UniqueName: \"kubernetes.io/projected/8a536ffc-bdda-43ee-bb29-cb07eba27ff8-kube-api-access-rgfmf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp\" (UID: \"8a536ffc-bdda-43ee-bb29-cb07eba27ff8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp"
Sep 10 15:49:06.924949 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.924893    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a536ffc-bdda-43ee-bb29-cb07eba27ff8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp\" (UID: \"8a536ffc-bdda-43ee-bb29-cb07eba27ff8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp"
Sep 10 15:49:06.924949 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.924919    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hqkz\" (UniqueName: \"kubernetes.io/projected/411b311a-46f3-441a-b8f1-0473409b058b-kube-api-access-2hqkz\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp\" (UID: \"411b311a-46f3-441a-b8f1-0473409b058b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp"
Sep 10 15:49:06.925011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:06.924976    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/411b311a-46f3-441a-b8f1-0473409b058b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp\" (UID: \"411b311a-46f3-441a-b8f1-0473409b058b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp"
Sep 10 15:49:07.005783 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.005699    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh"]
Sep 10 15:49:07.009231 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.009210    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh"
Sep 10 15:49:07.017481 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.017454    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh"]
Sep 10 15:49:07.025515 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.025492    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65"
Sep 10 15:49:07.025709 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.025692    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0de1915f-d4d4-4ec9-a378-f0ee550264ac-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh\" (UID: \"0de1915f-d4d4-4ec9-a378-f0ee550264ac\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh"
Sep 10 15:49:07.025757 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.025727    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a536ffc-bdda-43ee-bb29-cb07eba27ff8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp\" (UID: \"8a536ffc-bdda-43ee-bb29-cb07eba27ff8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp"
Sep 10 15:49:07.025757 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.025748    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqkz\" (UniqueName: \"kubernetes.io/projected/411b311a-46f3-441a-b8f1-0473409b058b-kube-api-access-2hqkz\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp\" (UID: \"411b311a-46f3-441a-b8f1-0473409b058b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp"
Sep 10 15:49:07.025836 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.025811    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/411b311a-46f3-441a-b8f1-0473409b058b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp\" (UID: \"411b311a-46f3-441a-b8f1-0473409b058b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp"
Sep 10 15:49:07.025872 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.025856    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0de1915f-d4d4-4ec9-a378-f0ee550264ac-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh\" (UID: \"0de1915f-d4d4-4ec9-a378-f0ee550264ac\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh"
Sep 10 15:49:07.025940 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.025926    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a536ffc-bdda-43ee-bb29-cb07eba27ff8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp\" (UID: \"8a536ffc-bdda-43ee-bb29-cb07eba27ff8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp"
Sep 10 15:49:07.025995 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.025973    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/411b311a-46f3-441a-b8f1-0473409b058b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp\" (UID: \"411b311a-46f3-441a-b8f1-0473409b058b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp"
Sep 10 15:49:07.026058 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.026019    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a536ffc-bdda-43ee-bb29-cb07eba27ff8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp\" (UID: \"8a536ffc-bdda-43ee-bb29-cb07eba27ff8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp"
Sep 10 15:49:07.026111 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.026090    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dlnf\" (UniqueName: \"kubernetes.io/projected/0de1915f-d4d4-4ec9-a378-f0ee550264ac-kube-api-access-9dlnf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh\" (UID: \"0de1915f-d4d4-4ec9-a378-f0ee550264ac\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh"
Sep 10 15:49:07.026167 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.026137    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgfmf\" (UniqueName: \"kubernetes.io/projected/8a536ffc-bdda-43ee-bb29-cb07eba27ff8-kube-api-access-rgfmf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp\" (UID: \"8a536ffc-bdda-43ee-bb29-cb07eba27ff8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp"
Sep 10 15:49:07.026295 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.026227    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/411b311a-46f3-441a-b8f1-0473409b058b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp\" (UID: \"411b311a-46f3-441a-b8f1-0473409b058b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp"
Sep 10 15:49:07.026295 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.026295    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/411b311a-46f3-441a-b8f1-0473409b058b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp\" (UID: \"411b311a-46f3-441a-b8f1-0473409b058b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp"
Sep 10 15:49:07.026434 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.026346    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a536ffc-bdda-43ee-bb29-cb07eba27ff8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp\" (UID: \"8a536ffc-bdda-43ee-bb29-cb07eba27ff8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp"
Sep 10 15:49:07.035610 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.035594    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgfmf\" (UniqueName: \"kubernetes.io/projected/8a536ffc-bdda-43ee-bb29-cb07eba27ff8-kube-api-access-rgfmf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp\" (UID: \"8a536ffc-bdda-43ee-bb29-cb07eba27ff8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp"
Sep 10 15:49:07.035770 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.035749    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hqkz\" (UniqueName: \"kubernetes.io/projected/411b311a-46f3-441a-b8f1-0473409b058b-kube-api-access-2hqkz\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp\" (UID: \"411b311a-46f3-441a-b8f1-0473409b058b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp"
Sep 10 15:49:07.120691 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.120650    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp"
Sep 10 15:49:07.128783 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.128754    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dlnf\" (UniqueName: \"kubernetes.io/projected/0de1915f-d4d4-4ec9-a378-f0ee550264ac-kube-api-access-9dlnf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh\" (UID: \"0de1915f-d4d4-4ec9-a378-f0ee550264ac\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh"
Sep 10 15:49:07.128936 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.128798    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0de1915f-d4d4-4ec9-a378-f0ee550264ac-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh\" (UID: \"0de1915f-d4d4-4ec9-a378-f0ee550264ac\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh"
Sep 10 15:49:07.128936 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.128844    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0de1915f-d4d4-4ec9-a378-f0ee550264ac-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh\" (UID: \"0de1915f-d4d4-4ec9-a378-f0ee550264ac\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh"
Sep 10 15:49:07.129176 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.129159    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0de1915f-d4d4-4ec9-a378-f0ee550264ac-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh\" (UID: \"0de1915f-d4d4-4ec9-a378-f0ee550264ac\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh"
Sep 10 15:49:07.129228 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.129204    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0de1915f-d4d4-4ec9-a378-f0ee550264ac-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh\" (UID: \"0de1915f-d4d4-4ec9-a378-f0ee550264ac\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh"
Sep 10 15:49:07.137883 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.137850    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dlnf\" (UniqueName: \"kubernetes.io/projected/0de1915f-d4d4-4ec9-a378-f0ee550264ac-kube-api-access-9dlnf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh\" (UID: \"0de1915f-d4d4-4ec9-a378-f0ee550264ac\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh"
Sep 10 15:49:07.221812 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.221782    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp"
Sep 10 15:49:07.299574 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.299537    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65"]
Sep 10 15:49:07.302648 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:49:07.302616    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469c7349_f352_4b2d_9db0_79cc0c05aa7a.slice/crio-d45470468a220435c2fd80ee82fea150b926662a921e6a6e5068ca6cb45fad5c WatchSource:0}: Error finding container d45470468a220435c2fd80ee82fea150b926662a921e6a6e5068ca6cb45fad5c: Status 404 returned error can't find the container with id d45470468a220435c2fd80ee82fea150b926662a921e6a6e5068ca6cb45fad5c
Sep 10 15:49:07.318393 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.318373    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh"
Sep 10 15:49:07.402761 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.402720    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp"]
Sep 10 15:49:07.403084 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:49:07.403055    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a536ffc_bdda_43ee_bb29_cb07eba27ff8.slice/crio-df5818e0be0ab006251a6dc385d0e78ff938fae8a43404d461e53bf0e19d22f2 WatchSource:0}: Error finding container df5818e0be0ab006251a6dc385d0e78ff938fae8a43404d461e53bf0e19d22f2: Status 404 returned error can't find the container with id df5818e0be0ab006251a6dc385d0e78ff938fae8a43404d461e53bf0e19d22f2
Sep 10 15:49:07.494979 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.494955    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp"]
Sep 10 15:49:07.496761 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:49:07.496735    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411b311a_46f3_441a_b8f1_0473409b058b.slice/crio-b6200c9faf2ad5ddceebc2280ad2b2c931c45b348354c4442d691b66e8bac3cc WatchSource:0}: Error finding container b6200c9faf2ad5ddceebc2280ad2b2c931c45b348354c4442d691b66e8bac3cc: Status 404 returned error can't find the container with id b6200c9faf2ad5ddceebc2280ad2b2c931c45b348354c4442d691b66e8bac3cc
Sep 10 15:49:07.597869 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.597841    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh"]
Sep 10 15:49:07.611947 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:49:07.611921    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de1915f_d4d4_4ec9_a378_f0ee550264ac.slice/crio-9e5c3e3ff7c91c2e41ad7dd2765966d0e00182025d75d690e0141d90481f86c1 WatchSource:0}: Error finding container 9e5c3e3ff7c91c2e41ad7dd2765966d0e00182025d75d690e0141d90481f86c1: Status 404 returned error can't find the container with id 9e5c3e3ff7c91c2e41ad7dd2765966d0e00182025d75d690e0141d90481f86c1
Sep 10 15:49:07.820486 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.820454    2572 generic.go:356] "Generic (PLEG): container finished" podID="411b311a-46f3-441a-b8f1-0473409b058b" containerID="f32c2915aa7bc916b7dd12c01083fd90c8efe189f9d07ef95df344dcfebb8dd8" exitCode=0
Sep 10 15:49:07.820901 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.820542    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp" event={"ID":"411b311a-46f3-441a-b8f1-0473409b058b","Type":"ContainerDied","Data":"f32c2915aa7bc916b7dd12c01083fd90c8efe189f9d07ef95df344dcfebb8dd8"}
Sep 10 15:49:07.820901 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.820629    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp" event={"ID":"411b311a-46f3-441a-b8f1-0473409b058b","Type":"ContainerStarted","Data":"b6200c9faf2ad5ddceebc2280ad2b2c931c45b348354c4442d691b66e8bac3cc"}
Sep 10 15:49:07.822266 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.822236    2572 generic.go:356] "Generic (PLEG): container finished" podID="0de1915f-d4d4-4ec9-a378-f0ee550264ac" containerID="afec4f9b7c65c508748699683b0ce3f93531d80770633fa0d7cfed6f0c2a1dd3" exitCode=0
Sep 10 15:49:07.822422 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.822399    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh" event={"ID":"0de1915f-d4d4-4ec9-a378-f0ee550264ac","Type":"ContainerDied","Data":"afec4f9b7c65c508748699683b0ce3f93531d80770633fa0d7cfed6f0c2a1dd3"}
Sep 10 15:49:07.822544 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.822446    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh" event={"ID":"0de1915f-d4d4-4ec9-a378-f0ee550264ac","Type":"ContainerStarted","Data":"9e5c3e3ff7c91c2e41ad7dd2765966d0e00182025d75d690e0141d90481f86c1"}
Sep 10 15:49:07.824047 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.824028    2572 generic.go:356] "Generic (PLEG): container finished" podID="8a536ffc-bdda-43ee-bb29-cb07eba27ff8" containerID="997128f31e9a81a8baae7ff55157d22b95f60bd033ae0fe2ae8058afe3ed083c" exitCode=0
Sep 10 15:49:07.824122 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.824093    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp" event={"ID":"8a536ffc-bdda-43ee-bb29-cb07eba27ff8","Type":"ContainerDied","Data":"997128f31e9a81a8baae7ff55157d22b95f60bd033ae0fe2ae8058afe3ed083c"}
Sep 10 15:49:07.824179 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.824134    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp" event={"ID":"8a536ffc-bdda-43ee-bb29-cb07eba27ff8","Type":"ContainerStarted","Data":"df5818e0be0ab006251a6dc385d0e78ff938fae8a43404d461e53bf0e19d22f2"}
Sep 10 15:49:07.825718 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.825696    2572 generic.go:356] "Generic (PLEG): container finished" podID="469c7349-f352-4b2d-9db0-79cc0c05aa7a" containerID="73f60475f040beb54cd0fdc9a0006d388e0742fd20d632356498cef91c433746" exitCode=0
Sep 10 15:49:07.825817 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.825743    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65" event={"ID":"469c7349-f352-4b2d-9db0-79cc0c05aa7a","Type":"ContainerDied","Data":"73f60475f040beb54cd0fdc9a0006d388e0742fd20d632356498cef91c433746"}
Sep 10 15:49:07.825817 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:07.825786    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65" event={"ID":"469c7349-f352-4b2d-9db0-79cc0c05aa7a","Type":"ContainerStarted","Data":"d45470468a220435c2fd80ee82fea150b926662a921e6a6e5068ca6cb45fad5c"}
Sep 10 15:49:09.838750 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:09.838711    2572 generic.go:356] "Generic (PLEG): container finished" podID="411b311a-46f3-441a-b8f1-0473409b058b" containerID="1af95e7d215c81ed74a7811f35651c1a345b6bbfeb2a2beda192a270e63b8081" exitCode=0
Sep 10 15:49:09.839211 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:09.838800    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp" event={"ID":"411b311a-46f3-441a-b8f1-0473409b058b","Type":"ContainerDied","Data":"1af95e7d215c81ed74a7811f35651c1a345b6bbfeb2a2beda192a270e63b8081"}
Sep 10 15:49:09.840869 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:09.840843    2572 generic.go:356] "Generic (PLEG): container finished" podID="0de1915f-d4d4-4ec9-a378-f0ee550264ac" containerID="11ef067d77c1ac59aaf663d6c3b34131675e3498c026d0cce97073208266166e" exitCode=0
Sep 10 15:49:09.840971 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:09.840931    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh" event={"ID":"0de1915f-d4d4-4ec9-a378-f0ee550264ac","Type":"ContainerDied","Data":"11ef067d77c1ac59aaf663d6c3b34131675e3498c026d0cce97073208266166e"}
Sep 10 15:49:09.842954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:09.842932    2572 generic.go:356] "Generic (PLEG): container finished" podID="8a536ffc-bdda-43ee-bb29-cb07eba27ff8" containerID="41eda67ae056f73b92a39c72e137472067d71c79e402510c49113f7a021e8fe9" exitCode=0
Sep 10 15:49:09.843039 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:09.843000    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp" event={"ID":"8a536ffc-bdda-43ee-bb29-cb07eba27ff8","Type":"ContainerDied","Data":"41eda67ae056f73b92a39c72e137472067d71c79e402510c49113f7a021e8fe9"}
Sep 10 15:49:09.845082 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:09.845060    2572 generic.go:356] "Generic (PLEG): container finished" podID="469c7349-f352-4b2d-9db0-79cc0c05aa7a" containerID="c1fc725a067d5e4e16502978a322be4d1cce8e12e83fa539f65a9e74408c3c12" exitCode=0
Sep 10 15:49:09.845163 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:09.845123    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65" event={"ID":"469c7349-f352-4b2d-9db0-79cc0c05aa7a","Type":"ContainerDied","Data":"c1fc725a067d5e4e16502978a322be4d1cce8e12e83fa539f65a9e74408c3c12"}
Sep 10 15:49:10.852087 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:10.852050    2572 generic.go:356] "Generic (PLEG): container finished" podID="8a536ffc-bdda-43ee-bb29-cb07eba27ff8" containerID="609ef2698b17a211c2f2fe29cc85b68fc9e7e27c37f8cce5fd710911dc3d994d" exitCode=0
Sep 10 15:49:10.852509 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:10.852138    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp" event={"ID":"8a536ffc-bdda-43ee-bb29-cb07eba27ff8","Type":"ContainerDied","Data":"609ef2698b17a211c2f2fe29cc85b68fc9e7e27c37f8cce5fd710911dc3d994d"}
Sep 10 15:49:10.854267 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:10.854235    2572 generic.go:356] "Generic (PLEG): container finished" podID="469c7349-f352-4b2d-9db0-79cc0c05aa7a" containerID="8836c8ac750b2cf10d79aac45da206fc95a663fbe20c29020ec5e3802813afde" exitCode=0
Sep 10 15:49:10.854395 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:10.854327    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65" event={"ID":"469c7349-f352-4b2d-9db0-79cc0c05aa7a","Type":"ContainerDied","Data":"8836c8ac750b2cf10d79aac45da206fc95a663fbe20c29020ec5e3802813afde"}
Sep 10 15:49:10.856326 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:10.856288    2572 generic.go:356] "Generic (PLEG): container finished" podID="411b311a-46f3-441a-b8f1-0473409b058b" containerID="80a3f63ae6500f768ab694f0b33831863b0d429297114c93d2289066bde6d2af" exitCode=0
Sep 10 15:49:10.856434 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:10.856333    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp" event={"ID":"411b311a-46f3-441a-b8f1-0473409b058b","Type":"ContainerDied","Data":"80a3f63ae6500f768ab694f0b33831863b0d429297114c93d2289066bde6d2af"}
Sep 10 15:49:10.858399 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:10.858377    2572 generic.go:356] "Generic (PLEG): container finished" podID="0de1915f-d4d4-4ec9-a378-f0ee550264ac" containerID="46488ab69ab044c26fb57c2771c48c87520b5005a2e97e593943fcfb37c21916" exitCode=0
Sep 10 15:49:10.858495 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:10.858435    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh" event={"ID":"0de1915f-d4d4-4ec9-a378-f0ee550264ac","Type":"ContainerDied","Data":"46488ab69ab044c26fb57c2771c48c87520b5005a2e97e593943fcfb37c21916"}
Sep 10 15:49:12.030727 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.030697    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp"
Sep 10 15:49:12.033010 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.032986    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh"
Sep 10 15:49:12.060335 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.060294    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a536ffc-bdda-43ee-bb29-cb07eba27ff8-util\") pod \"8a536ffc-bdda-43ee-bb29-cb07eba27ff8\" (UID: \"8a536ffc-bdda-43ee-bb29-cb07eba27ff8\") "
Sep 10 15:49:12.060455 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.060353    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0de1915f-d4d4-4ec9-a378-f0ee550264ac-bundle\") pod \"0de1915f-d4d4-4ec9-a378-f0ee550264ac\" (UID: \"0de1915f-d4d4-4ec9-a378-f0ee550264ac\") "
Sep 10 15:49:12.060455 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.060384    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dlnf\" (UniqueName: \"kubernetes.io/projected/0de1915f-d4d4-4ec9-a378-f0ee550264ac-kube-api-access-9dlnf\") pod \"0de1915f-d4d4-4ec9-a378-f0ee550264ac\" (UID: \"0de1915f-d4d4-4ec9-a378-f0ee550264ac\") "
Sep 10 15:49:12.060455 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.060434    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0de1915f-d4d4-4ec9-a378-f0ee550264ac-util\") pod \"0de1915f-d4d4-4ec9-a378-f0ee550264ac\" (UID: \"0de1915f-d4d4-4ec9-a378-f0ee550264ac\") "
Sep 10 15:49:12.060676 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.060489    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a536ffc-bdda-43ee-bb29-cb07eba27ff8-bundle\") pod \"8a536ffc-bdda-43ee-bb29-cb07eba27ff8\" (UID: \"8a536ffc-bdda-43ee-bb29-cb07eba27ff8\") "
Sep 10 15:49:12.060676 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.060524    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgfmf\" (UniqueName: \"kubernetes.io/projected/8a536ffc-bdda-43ee-bb29-cb07eba27ff8-kube-api-access-rgfmf\") pod \"8a536ffc-bdda-43ee-bb29-cb07eba27ff8\" (UID: \"8a536ffc-bdda-43ee-bb29-cb07eba27ff8\") "
Sep 10 15:49:12.061445 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.061362    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp"
Sep 10 15:49:12.061445 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.061400    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de1915f-d4d4-4ec9-a378-f0ee550264ac-bundle" (OuterVolumeSpecName: "bundle") pod "0de1915f-d4d4-4ec9-a378-f0ee550264ac" (UID: "0de1915f-d4d4-4ec9-a378-f0ee550264ac"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:49:12.061650 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.061623    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a536ffc-bdda-43ee-bb29-cb07eba27ff8-bundle" (OuterVolumeSpecName: "bundle") pod "8a536ffc-bdda-43ee-bb29-cb07eba27ff8" (UID: "8a536ffc-bdda-43ee-bb29-cb07eba27ff8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:49:12.063283 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.063247    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de1915f-d4d4-4ec9-a378-f0ee550264ac-kube-api-access-9dlnf" (OuterVolumeSpecName: "kube-api-access-9dlnf") pod "0de1915f-d4d4-4ec9-a378-f0ee550264ac" (UID: "0de1915f-d4d4-4ec9-a378-f0ee550264ac"). InnerVolumeSpecName "kube-api-access-9dlnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:49:12.063942 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.063910    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a536ffc-bdda-43ee-bb29-cb07eba27ff8-kube-api-access-rgfmf" (OuterVolumeSpecName: "kube-api-access-rgfmf") pod "8a536ffc-bdda-43ee-bb29-cb07eba27ff8" (UID: "8a536ffc-bdda-43ee-bb29-cb07eba27ff8"). InnerVolumeSpecName "kube-api-access-rgfmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:49:12.066872 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.066851    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65"
Sep 10 15:49:12.067107 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.067084    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a536ffc-bdda-43ee-bb29-cb07eba27ff8-util" (OuterVolumeSpecName: "util") pod "8a536ffc-bdda-43ee-bb29-cb07eba27ff8" (UID: "8a536ffc-bdda-43ee-bb29-cb07eba27ff8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:49:12.067431 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.067412    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de1915f-d4d4-4ec9-a378-f0ee550264ac-util" (OuterVolumeSpecName: "util") pod "0de1915f-d4d4-4ec9-a378-f0ee550264ac" (UID: "0de1915f-d4d4-4ec9-a378-f0ee550264ac"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:49:12.161594 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.161505    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/469c7349-f352-4b2d-9db0-79cc0c05aa7a-bundle\") pod \"469c7349-f352-4b2d-9db0-79cc0c05aa7a\" (UID: \"469c7349-f352-4b2d-9db0-79cc0c05aa7a\") "
Sep 10 15:49:12.161594 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.161544    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/411b311a-46f3-441a-b8f1-0473409b058b-bundle\") pod \"411b311a-46f3-441a-b8f1-0473409b058b\" (UID: \"411b311a-46f3-441a-b8f1-0473409b058b\") "
Sep 10 15:49:12.161594 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.161562    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/411b311a-46f3-441a-b8f1-0473409b058b-util\") pod \"411b311a-46f3-441a-b8f1-0473409b058b\" (UID: \"411b311a-46f3-441a-b8f1-0473409b058b\") "
Sep 10 15:49:12.161594 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.161583    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hqkz\" (UniqueName: \"kubernetes.io/projected/411b311a-46f3-441a-b8f1-0473409b058b-kube-api-access-2hqkz\") pod \"411b311a-46f3-441a-b8f1-0473409b058b\" (UID: \"411b311a-46f3-441a-b8f1-0473409b058b\") "
Sep 10 15:49:12.161929 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.161611    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/469c7349-f352-4b2d-9db0-79cc0c05aa7a-util\") pod \"469c7349-f352-4b2d-9db0-79cc0c05aa7a\" (UID: \"469c7349-f352-4b2d-9db0-79cc0c05aa7a\") "
Sep 10 15:49:12.161929 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.161698    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n66fj\" (UniqueName: \"kubernetes.io/projected/469c7349-f352-4b2d-9db0-79cc0c05aa7a-kube-api-access-n66fj\") pod \"469c7349-f352-4b2d-9db0-79cc0c05aa7a\" (UID: \"469c7349-f352-4b2d-9db0-79cc0c05aa7a\") "
Sep 10 15:49:12.161929 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.161922    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a536ffc-bdda-43ee-bb29-cb07eba27ff8-util\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:49:12.162086 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.161938    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0de1915f-d4d4-4ec9-a378-f0ee550264ac-bundle\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:49:12.162086 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.161951    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dlnf\" (UniqueName: \"kubernetes.io/projected/0de1915f-d4d4-4ec9-a378-f0ee550264ac-kube-api-access-9dlnf\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:49:12.162086 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.161963    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0de1915f-d4d4-4ec9-a378-f0ee550264ac-util\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:49:12.162086 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.161977    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a536ffc-bdda-43ee-bb29-cb07eba27ff8-bundle\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:49:12.162086 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.161994    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgfmf\" (UniqueName: \"kubernetes.io/projected/8a536ffc-bdda-43ee-bb29-cb07eba27ff8-kube-api-access-rgfmf\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:49:12.162350 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.162171    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/411b311a-46f3-441a-b8f1-0473409b058b-bundle" (OuterVolumeSpecName: "bundle") pod "411b311a-46f3-441a-b8f1-0473409b058b" (UID: "411b311a-46f3-441a-b8f1-0473409b058b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:49:12.162396 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.162347    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469c7349-f352-4b2d-9db0-79cc0c05aa7a-bundle" (OuterVolumeSpecName: "bundle") pod "469c7349-f352-4b2d-9db0-79cc0c05aa7a" (UID: "469c7349-f352-4b2d-9db0-79cc0c05aa7a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:49:12.163827 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.163800    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411b311a-46f3-441a-b8f1-0473409b058b-kube-api-access-2hqkz" (OuterVolumeSpecName: "kube-api-access-2hqkz") pod "411b311a-46f3-441a-b8f1-0473409b058b" (UID: "411b311a-46f3-441a-b8f1-0473409b058b"). InnerVolumeSpecName "kube-api-access-2hqkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:49:12.164090 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.164072    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469c7349-f352-4b2d-9db0-79cc0c05aa7a-kube-api-access-n66fj" (OuterVolumeSpecName: "kube-api-access-n66fj") pod "469c7349-f352-4b2d-9db0-79cc0c05aa7a" (UID: "469c7349-f352-4b2d-9db0-79cc0c05aa7a"). InnerVolumeSpecName "kube-api-access-n66fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:49:12.167852 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.167830    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469c7349-f352-4b2d-9db0-79cc0c05aa7a-util" (OuterVolumeSpecName: "util") pod "469c7349-f352-4b2d-9db0-79cc0c05aa7a" (UID: "469c7349-f352-4b2d-9db0-79cc0c05aa7a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:49:12.168281 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.168261    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/411b311a-46f3-441a-b8f1-0473409b058b-util" (OuterVolumeSpecName: "util") pod "411b311a-46f3-441a-b8f1-0473409b058b" (UID: "411b311a-46f3-441a-b8f1-0473409b058b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:49:12.262547 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.262510    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n66fj\" (UniqueName: \"kubernetes.io/projected/469c7349-f352-4b2d-9db0-79cc0c05aa7a-kube-api-access-n66fj\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:49:12.262709 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.262566    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/469c7349-f352-4b2d-9db0-79cc0c05aa7a-bundle\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:49:12.262709 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.262580    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/411b311a-46f3-441a-b8f1-0473409b058b-bundle\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:49:12.262709 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.262592    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/411b311a-46f3-441a-b8f1-0473409b058b-util\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:49:12.262709 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.262604    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hqkz\" (UniqueName: \"kubernetes.io/projected/411b311a-46f3-441a-b8f1-0473409b058b-kube-api-access-2hqkz\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:49:12.262709 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.262615    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/469c7349-f352-4b2d-9db0-79cc0c05aa7a-util\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:49:12.869841 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.869816    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp"
Sep 10 15:49:12.870020 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.869815    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adv2tp" event={"ID":"8a536ffc-bdda-43ee-bb29-cb07eba27ff8","Type":"ContainerDied","Data":"df5818e0be0ab006251a6dc385d0e78ff938fae8a43404d461e53bf0e19d22f2"}
Sep 10 15:49:12.870020 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.869938    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df5818e0be0ab006251a6dc385d0e78ff938fae8a43404d461e53bf0e19d22f2"
Sep 10 15:49:12.871930 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.871910    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65"
Sep 10 15:49:12.872047 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.871947    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6r8r65" event={"ID":"469c7349-f352-4b2d-9db0-79cc0c05aa7a","Type":"ContainerDied","Data":"d45470468a220435c2fd80ee82fea150b926662a921e6a6e5068ca6cb45fad5c"}
Sep 10 15:49:12.872047 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.871992    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d45470468a220435c2fd80ee82fea150b926662a921e6a6e5068ca6cb45fad5c"
Sep 10 15:49:12.873855 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.873836    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp"
Sep 10 15:49:12.873980 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.873865    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2rdlp" event={"ID":"411b311a-46f3-441a-b8f1-0473409b058b","Type":"ContainerDied","Data":"b6200c9faf2ad5ddceebc2280ad2b2c931c45b348354c4442d691b66e8bac3cc"}
Sep 10 15:49:12.873980 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.873914    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6200c9faf2ad5ddceebc2280ad2b2c931c45b348354c4442d691b66e8bac3cc"
Sep 10 15:49:12.876384 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.876361    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh" event={"ID":"0de1915f-d4d4-4ec9-a378-f0ee550264ac","Type":"ContainerDied","Data":"9e5c3e3ff7c91c2e41ad7dd2765966d0e00182025d75d690e0141d90481f86c1"}
Sep 10 15:49:12.876485 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.876393    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7snbnh"
Sep 10 15:49:12.876485 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:12.876400    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e5c3e3ff7c91c2e41ad7dd2765966d0e00182025d75d690e0141d90481f86c1"
Sep 10 15:49:28.265538 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265504    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-r4gbw"]
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265811    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="411b311a-46f3-441a-b8f1-0473409b058b" containerName="extract"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265822    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="411b311a-46f3-441a-b8f1-0473409b058b" containerName="extract"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265836    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="469c7349-f352-4b2d-9db0-79cc0c05aa7a" containerName="util"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265841    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="469c7349-f352-4b2d-9db0-79cc0c05aa7a" containerName="util"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265847    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="411b311a-46f3-441a-b8f1-0473409b058b" containerName="util"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265853    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="411b311a-46f3-441a-b8f1-0473409b058b" containerName="util"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265860    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="411b311a-46f3-441a-b8f1-0473409b058b" containerName="pull"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265866    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="411b311a-46f3-441a-b8f1-0473409b058b" containerName="pull"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265873    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a536ffc-bdda-43ee-bb29-cb07eba27ff8" containerName="extract"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265878    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a536ffc-bdda-43ee-bb29-cb07eba27ff8" containerName="extract"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265886    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="469c7349-f352-4b2d-9db0-79cc0c05aa7a" containerName="extract"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265891    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="469c7349-f352-4b2d-9db0-79cc0c05aa7a" containerName="extract"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265899    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0de1915f-d4d4-4ec9-a378-f0ee550264ac" containerName="pull"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265903    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0de1915f-d4d4-4ec9-a378-f0ee550264ac" containerName="pull"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265911    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0de1915f-d4d4-4ec9-a378-f0ee550264ac" containerName="extract"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265918    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0de1915f-d4d4-4ec9-a378-f0ee550264ac" containerName="extract"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265928    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a536ffc-bdda-43ee-bb29-cb07eba27ff8" containerName="util"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265936    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a536ffc-bdda-43ee-bb29-cb07eba27ff8" containerName="util"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265944    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0de1915f-d4d4-4ec9-a378-f0ee550264ac" containerName="util"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265951    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0de1915f-d4d4-4ec9-a378-f0ee550264ac" containerName="util"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265958    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="469c7349-f352-4b2d-9db0-79cc0c05aa7a" containerName="pull"
Sep 10 15:49:28.265954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265966    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="469c7349-f352-4b2d-9db0-79cc0c05aa7a" containerName="pull"
Sep 10 15:49:28.266632 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265982    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a536ffc-bdda-43ee-bb29-cb07eba27ff8" containerName="pull"
Sep 10 15:49:28.266632 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.265990    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a536ffc-bdda-43ee-bb29-cb07eba27ff8" containerName="pull"
Sep 10 15:49:28.266632 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.266047    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="411b311a-46f3-441a-b8f1-0473409b058b" containerName="extract"
Sep 10 15:49:28.266632 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.266056    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0de1915f-d4d4-4ec9-a378-f0ee550264ac" containerName="extract"
Sep 10 15:49:28.266632 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.266063    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="469c7349-f352-4b2d-9db0-79cc0c05aa7a" containerName="extract"
Sep 10 15:49:28.266632 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.266068    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a536ffc-bdda-43ee-bb29-cb07eba27ff8" containerName="extract"
Sep 10 15:49:28.270373 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.270350    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r4gbw"
Sep 10 15:49:28.273963 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.273940    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 15:49:28.274090 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.273939    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pj7x6\""
Sep 10 15:49:28.274090 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.274034    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 15:49:28.289295 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.289267    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 15:49:28.289295 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.289267    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 15:49:28.310753 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.310720    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-r4gbw"]
Sep 10 15:49:28.391635 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.391599    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j64rz\" (UniqueName: \"kubernetes.io/projected/5f160b88-c477-41cf-beed-662a1f108f8d-kube-api-access-j64rz\") pod \"kuadrant-console-plugin-fbdd4f445-r4gbw\" (UID: \"5f160b88-c477-41cf-beed-662a1f108f8d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r4gbw"
Sep 10 15:49:28.391635 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.391635    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f160b88-c477-41cf-beed-662a1f108f8d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-r4gbw\" (UID: \"5f160b88-c477-41cf-beed-662a1f108f8d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r4gbw"
Sep 10 15:49:28.391850 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.391697    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5f160b88-c477-41cf-beed-662a1f108f8d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-r4gbw\" (UID: \"5f160b88-c477-41cf-beed-662a1f108f8d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r4gbw"
Sep 10 15:49:28.493171 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.493130    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j64rz\" (UniqueName: \"kubernetes.io/projected/5f160b88-c477-41cf-beed-662a1f108f8d-kube-api-access-j64rz\") pod \"kuadrant-console-plugin-fbdd4f445-r4gbw\" (UID: \"5f160b88-c477-41cf-beed-662a1f108f8d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r4gbw"
Sep 10 15:49:28.493171 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.493176    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f160b88-c477-41cf-beed-662a1f108f8d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-r4gbw\" (UID: \"5f160b88-c477-41cf-beed-662a1f108f8d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r4gbw"
Sep 10 15:49:28.493427 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.493241    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5f160b88-c477-41cf-beed-662a1f108f8d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-r4gbw\" (UID: \"5f160b88-c477-41cf-beed-662a1f108f8d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r4gbw"
Sep 10 15:49:28.493981 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.493954    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5f160b88-c477-41cf-beed-662a1f108f8d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-r4gbw\" (UID: \"5f160b88-c477-41cf-beed-662a1f108f8d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r4gbw"
Sep 10 15:49:28.495730 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.495711    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f160b88-c477-41cf-beed-662a1f108f8d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-r4gbw\" (UID: \"5f160b88-c477-41cf-beed-662a1f108f8d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r4gbw"
Sep 10 15:49:28.535057 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.534972    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j64rz\" (UniqueName: \"kubernetes.io/projected/5f160b88-c477-41cf-beed-662a1f108f8d-kube-api-access-j64rz\") pod \"kuadrant-console-plugin-fbdd4f445-r4gbw\" (UID: \"5f160b88-c477-41cf-beed-662a1f108f8d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r4gbw"
Sep 10 15:49:28.580019 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.579976    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r4gbw"
Sep 10 15:49:28.855420 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.855392    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-r4gbw"]
Sep 10 15:49:28.857843 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:49:28.857814    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f160b88_c477_41cf_beed_662a1f108f8d.slice/crio-fb06f69ee24007632eac4f7ea3abc8c348fe6ba8758c2aee22ddfb9105036b16 WatchSource:0}: Error finding container fb06f69ee24007632eac4f7ea3abc8c348fe6ba8758c2aee22ddfb9105036b16: Status 404 returned error can't find the container with id fb06f69ee24007632eac4f7ea3abc8c348fe6ba8758c2aee22ddfb9105036b16
Sep 10 15:49:28.957630 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:28.957592    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r4gbw" event={"ID":"5f160b88-c477-41cf-beed-662a1f108f8d","Type":"ContainerStarted","Data":"fb06f69ee24007632eac4f7ea3abc8c348fe6ba8758c2aee22ddfb9105036b16"}
Sep 10 15:49:33.991817 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:33.991784    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r4gbw" event={"ID":"5f160b88-c477-41cf-beed-662a1f108f8d","Type":"ContainerStarted","Data":"8d2e0ed298a0e74354064fb870590ce7678d06a1121eb20463fda64703004bbb"}
Sep 10 15:49:34.005586 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:34.005521    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r4gbw" podStartSLOduration=1.535960435 podStartE2EDuration="6.005498661s" totalImagesPullingTime="4.469538226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:34.005447682 +0000 UTC m=+446.677857764" watchObservedRunningTime="2026-09-10 15:49:34.005498661 +0000 UTC m=+446.677908744"
Sep 10 15:49:40.416609 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:40.416572    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k"]
Sep 10 15:49:40.421147 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:40.421121    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k"
Sep 10 15:49:40.424975 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:40.424956    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5ls97\""
Sep 10 15:49:40.450152 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:40.450116    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k"]
Sep 10 15:49:40.500057 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:40.500013    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vflj2\" (UniqueName: \"kubernetes.io/projected/c3a4eb26-1798-4e4e-a16e-d76c441bf883-kube-api-access-vflj2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p428k\" (UID: \"c3a4eb26-1798-4e4e-a16e-d76c441bf883\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k"
Sep 10 15:49:40.500236 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:40.500073    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3a4eb26-1798-4e4e-a16e-d76c441bf883-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p428k\" (UID: \"c3a4eb26-1798-4e4e-a16e-d76c441bf883\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k"
Sep 10 15:49:40.600727 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:40.600688    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vflj2\" (UniqueName: \"kubernetes.io/projected/c3a4eb26-1798-4e4e-a16e-d76c441bf883-kube-api-access-vflj2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p428k\" (UID: \"c3a4eb26-1798-4e4e-a16e-d76c441bf883\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k"
Sep 10 15:49:40.600907 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:40.600757    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3a4eb26-1798-4e4e-a16e-d76c441bf883-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p428k\" (UID: \"c3a4eb26-1798-4e4e-a16e-d76c441bf883\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k"
Sep 10 15:49:40.601103 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:40.601082    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3a4eb26-1798-4e4e-a16e-d76c441bf883-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p428k\" (UID: \"c3a4eb26-1798-4e4e-a16e-d76c441bf883\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k"
Sep 10 15:49:40.610695 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:40.610656    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vflj2\" (UniqueName: \"kubernetes.io/projected/c3a4eb26-1798-4e4e-a16e-d76c441bf883-kube-api-access-vflj2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-p428k\" (UID: \"c3a4eb26-1798-4e4e-a16e-d76c441bf883\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k"
Sep 10 15:49:40.731734 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:40.731641    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k"
Sep 10 15:49:41.022334 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:41.022228    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k"]
Sep 10 15:49:41.027447 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:49:41.027404    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a4eb26_1798_4e4e_a16e_d76c441bf883.slice/crio-2f9d11f2017d5a7c302d56a1978a932e61345da843e70ff3dca5f100c6411e81 WatchSource:0}: Error finding container 2f9d11f2017d5a7c302d56a1978a932e61345da843e70ff3dca5f100c6411e81: Status 404 returned error can't find the container with id 2f9d11f2017d5a7c302d56a1978a932e61345da843e70ff3dca5f100c6411e81
Sep 10 15:49:42.030231 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:42.030154    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k" event={"ID":"c3a4eb26-1798-4e4e-a16e-d76c441bf883","Type":"ContainerStarted","Data":"2f9d11f2017d5a7c302d56a1978a932e61345da843e70ff3dca5f100c6411e81"}
Sep 10 15:49:45.048553 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:45.048520    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k" event={"ID":"c3a4eb26-1798-4e4e-a16e-d76c441bf883","Type":"ContainerStarted","Data":"e851d5250d0d2acba7afdae5c5123c2835fe9126c48e2e48edaae1887c25cdd7"}
Sep 10 15:49:45.048942 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:45.048643    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k"
Sep 10 15:49:45.074375 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:45.074316    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k" podStartSLOduration=1.170156505 podStartE2EDuration="5.074287439s" totalImagesPullingTime="3.904130934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:45.070905191 +0000 UTC m=+457.743315274" watchObservedRunningTime="2026-09-10 15:49:45.074287439 +0000 UTC m=+457.746697516"
Sep 10 15:49:56.056593 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:49:56.056558    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k"
Sep 10 15:50:15.219472 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:15.219432    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-5jfgl"]
Sep 10 15:50:15.237074 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:15.237039    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-5jfgl"]
Sep 10 15:50:15.237249 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:15.237134    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-5jfgl"
Sep 10 15:50:15.239283 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:15.239253    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mhkgb\""
Sep 10 15:50:15.288266 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:15.288229    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fvxr\" (UniqueName: \"kubernetes.io/projected/a5ddcdb3-40da-4c89-ad02-6eb95697f766-kube-api-access-6fvxr\") pod \"authorino-7bcfd8f9cd-5jfgl\" (UID: \"a5ddcdb3-40da-4c89-ad02-6eb95697f766\") " pod="kuadrant-system/authorino-7bcfd8f9cd-5jfgl"
Sep 10 15:50:15.358297 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:15.358240    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-cc5js"]
Sep 10 15:50:15.360514 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:15.360488    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cc5js"
Sep 10 15:50:15.374051 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:15.374020    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cc5js"]
Sep 10 15:50:15.388787 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:15.388754    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh8tr\" (UniqueName: \"kubernetes.io/projected/8683732e-50a1-46ee-96b3-3128a74dc812-kube-api-access-vh8tr\") pod \"authorino-59c974bd48-cc5js\" (UID: \"8683732e-50a1-46ee-96b3-3128a74dc812\") " pod="kuadrant-system/authorino-59c974bd48-cc5js"
Sep 10 15:50:15.388955 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:15.388836    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fvxr\" (UniqueName: \"kubernetes.io/projected/a5ddcdb3-40da-4c89-ad02-6eb95697f766-kube-api-access-6fvxr\") pod \"authorino-7bcfd8f9cd-5jfgl\" (UID: \"a5ddcdb3-40da-4c89-ad02-6eb95697f766\") " pod="kuadrant-system/authorino-7bcfd8f9cd-5jfgl"
Sep 10 15:50:15.397792 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:15.397765    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fvxr\" (UniqueName: \"kubernetes.io/projected/a5ddcdb3-40da-4c89-ad02-6eb95697f766-kube-api-access-6fvxr\") pod \"authorino-7bcfd8f9cd-5jfgl\" (UID: \"a5ddcdb3-40da-4c89-ad02-6eb95697f766\") " pod="kuadrant-system/authorino-7bcfd8f9cd-5jfgl"
Sep 10 15:50:15.490316 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:15.490209    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh8tr\" (UniqueName: \"kubernetes.io/projected/8683732e-50a1-46ee-96b3-3128a74dc812-kube-api-access-vh8tr\") pod \"authorino-59c974bd48-cc5js\" (UID: \"8683732e-50a1-46ee-96b3-3128a74dc812\") " pod="kuadrant-system/authorino-59c974bd48-cc5js"
Sep 10 15:50:15.497741 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:15.497713    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh8tr\" (UniqueName: \"kubernetes.io/projected/8683732e-50a1-46ee-96b3-3128a74dc812-kube-api-access-vh8tr\") pod \"authorino-59c974bd48-cc5js\" (UID: \"8683732e-50a1-46ee-96b3-3128a74dc812\") " pod="kuadrant-system/authorino-59c974bd48-cc5js"
Sep 10 15:50:15.547566 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:15.547526    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-5jfgl"
Sep 10 15:50:15.669796 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:15.669754    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cc5js"
Sep 10 15:50:15.816011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:15.815982    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-5jfgl"]
Sep 10 15:50:15.818224 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:50:15.818190    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5ddcdb3_40da_4c89_ad02_6eb95697f766.slice/crio-8f6847b2acd327764d356f08b4211224a03f40d5a51fc4f91c2ec95d446401e7 WatchSource:0}: Error finding container 8f6847b2acd327764d356f08b4211224a03f40d5a51fc4f91c2ec95d446401e7: Status 404 returned error can't find the container with id 8f6847b2acd327764d356f08b4211224a03f40d5a51fc4f91c2ec95d446401e7
Sep 10 15:50:15.942779 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:15.942749    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cc5js"]
Sep 10 15:50:15.944590 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:50:15.944557    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8683732e_50a1_46ee_96b3_3128a74dc812.slice/crio-30096cb6f078833523c6aa45370df62377ecbc7bb70d69dd9ee2c612bd77e699 WatchSource:0}: Error finding container 30096cb6f078833523c6aa45370df62377ecbc7bb70d69dd9ee2c612bd77e699: Status 404 returned error can't find the container with id 30096cb6f078833523c6aa45370df62377ecbc7bb70d69dd9ee2c612bd77e699
Sep 10 15:50:16.206820 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:16.206705    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-5jfgl" event={"ID":"a5ddcdb3-40da-4c89-ad02-6eb95697f766","Type":"ContainerStarted","Data":"8f6847b2acd327764d356f08b4211224a03f40d5a51fc4f91c2ec95d446401e7"}
Sep 10 15:50:16.208038 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:16.208011    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cc5js" event={"ID":"8683732e-50a1-46ee-96b3-3128a74dc812","Type":"ContainerStarted","Data":"30096cb6f078833523c6aa45370df62377ecbc7bb70d69dd9ee2c612bd77e699"}
Sep 10 15:50:18.222627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:18.222581    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cc5js" event={"ID":"8683732e-50a1-46ee-96b3-3128a74dc812","Type":"ContainerStarted","Data":"3717ab938047bada1772872686dd5b3221347f8af41678d75c040209d62ea31b"}
Sep 10 15:50:18.224768 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:18.224734    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-5jfgl" event={"ID":"a5ddcdb3-40da-4c89-ad02-6eb95697f766","Type":"ContainerStarted","Data":"e631c12d6b5bfa9f23c33eadc1bfc6cfdf6e59d12f30ab88aff801fb4378e53b"}
Sep 10 15:50:18.268080 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:18.268023    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-5jfgl" podStartSLOduration=0.998953676 podStartE2EDuration="3.268003086s" totalImagesPullingTime="2.26904941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:18.266434736 +0000 UTC m=+490.938844820" watchObservedRunningTime="2026-09-10 15:50:18.268003086 +0000 UTC m=+490.940413169"
Sep 10 15:50:18.268370 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:18.268344    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-cc5js" podStartSLOduration=1.126342494 podStartE2EDuration="3.268338343s" totalImagesPullingTime="2.141995849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:18.249551932 +0000 UTC m=+490.921962017" watchObservedRunningTime="2026-09-10 15:50:18.268338343 +0000 UTC m=+490.940748423"
Sep 10 15:50:18.309195 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:18.309155    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-5jfgl"]
Sep 10 15:50:20.234213 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:20.234166    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-5jfgl" podUID="a5ddcdb3-40da-4c89-ad02-6eb95697f766" containerName="authorino" containerID="cri-o://e631c12d6b5bfa9f23c33eadc1bfc6cfdf6e59d12f30ab88aff801fb4378e53b" gracePeriod=30
Sep 10 15:50:20.478357 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:20.478328    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-5jfgl"
Sep 10 15:50:20.534994 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:20.534891    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fvxr\" (UniqueName: \"kubernetes.io/projected/a5ddcdb3-40da-4c89-ad02-6eb95697f766-kube-api-access-6fvxr\") pod \"a5ddcdb3-40da-4c89-ad02-6eb95697f766\" (UID: \"a5ddcdb3-40da-4c89-ad02-6eb95697f766\") "
Sep 10 15:50:20.537171 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:20.537134    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ddcdb3-40da-4c89-ad02-6eb95697f766-kube-api-access-6fvxr" (OuterVolumeSpecName: "kube-api-access-6fvxr") pod "a5ddcdb3-40da-4c89-ad02-6eb95697f766" (UID: "a5ddcdb3-40da-4c89-ad02-6eb95697f766"). InnerVolumeSpecName "kube-api-access-6fvxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:20.635548 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:20.635501    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fvxr\" (UniqueName: \"kubernetes.io/projected/a5ddcdb3-40da-4c89-ad02-6eb95697f766-kube-api-access-6fvxr\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:50:21.241057 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:21.241020    2572 generic.go:356] "Generic (PLEG): container finished" podID="a5ddcdb3-40da-4c89-ad02-6eb95697f766" containerID="e631c12d6b5bfa9f23c33eadc1bfc6cfdf6e59d12f30ab88aff801fb4378e53b" exitCode=0
Sep 10 15:50:21.241502 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:21.241064    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-5jfgl"
Sep 10 15:50:21.241502 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:21.241102    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-5jfgl" event={"ID":"a5ddcdb3-40da-4c89-ad02-6eb95697f766","Type":"ContainerDied","Data":"e631c12d6b5bfa9f23c33eadc1bfc6cfdf6e59d12f30ab88aff801fb4378e53b"}
Sep 10 15:50:21.241502 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:21.241142    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-5jfgl" event={"ID":"a5ddcdb3-40da-4c89-ad02-6eb95697f766","Type":"ContainerDied","Data":"8f6847b2acd327764d356f08b4211224a03f40d5a51fc4f91c2ec95d446401e7"}
Sep 10 15:50:21.241502 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:21.241157    2572 scope.go:117] "RemoveContainer" containerID="e631c12d6b5bfa9f23c33eadc1bfc6cfdf6e59d12f30ab88aff801fb4378e53b"
Sep 10 15:50:21.250792 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:21.250771    2572 scope.go:117] "RemoveContainer" containerID="e631c12d6b5bfa9f23c33eadc1bfc6cfdf6e59d12f30ab88aff801fb4378e53b"
Sep 10 15:50:21.251158 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:50:21.251137    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e631c12d6b5bfa9f23c33eadc1bfc6cfdf6e59d12f30ab88aff801fb4378e53b\": container with ID starting with e631c12d6b5bfa9f23c33eadc1bfc6cfdf6e59d12f30ab88aff801fb4378e53b not found: ID does not exist" containerID="e631c12d6b5bfa9f23c33eadc1bfc6cfdf6e59d12f30ab88aff801fb4378e53b"
Sep 10 15:50:21.251215 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:21.251167    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e631c12d6b5bfa9f23c33eadc1bfc6cfdf6e59d12f30ab88aff801fb4378e53b"} err="failed to get container status \"e631c12d6b5bfa9f23c33eadc1bfc6cfdf6e59d12f30ab88aff801fb4378e53b\": rpc error: code = NotFound desc = could not find container \"e631c12d6b5bfa9f23c33eadc1bfc6cfdf6e59d12f30ab88aff801fb4378e53b\": container with ID starting with e631c12d6b5bfa9f23c33eadc1bfc6cfdf6e59d12f30ab88aff801fb4378e53b not found: ID does not exist"
Sep 10 15:50:21.264793 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:21.264761    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-5jfgl"]
Sep 10 15:50:21.268107 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:21.268077    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-5jfgl"]
Sep 10 15:50:21.991798 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:21.991764    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5ddcdb3-40da-4c89-ad02-6eb95697f766" path="/var/lib/kubelet/pods/a5ddcdb3-40da-4c89-ad02-6eb95697f766/volumes"
Sep 10 15:50:43.913990 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:43.913951    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-8jrhr"]
Sep 10 15:50:43.914403 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:43.914276    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ddcdb3-40da-4c89-ad02-6eb95697f766" containerName="authorino"
Sep 10 15:50:43.914403 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:43.914288    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ddcdb3-40da-4c89-ad02-6eb95697f766" containerName="authorino"
Sep 10 15:50:43.914403 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:43.914368    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5ddcdb3-40da-4c89-ad02-6eb95697f766" containerName="authorino"
Sep 10 15:50:43.917159 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:43.917139    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-8jrhr"
Sep 10 15:50:43.928792 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:43.928762    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-8jrhr"]
Sep 10 15:50:43.983378 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:43.983337    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-8jrhr"]
Sep 10 15:50:43.983629 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:50:43.983606    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-chwcp], unattached volumes=[], failed to process volumes=[kube-api-access-chwcp]: context canceled" pod="kuadrant-system/authorino-557f876f69-8jrhr" podUID="808b9b44-8c2d-4e08-a110-e6660181bea3"
Sep 10 15:50:44.010122 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.010087    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-q524n"]
Sep 10 15:50:44.013215 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.013193    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-q524n"
Sep 10 15:50:44.015250 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.015232    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 15:50:44.025948 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.021733    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-q524n"]
Sep 10 15:50:44.027622 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.027585    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chwcp\" (UniqueName: \"kubernetes.io/projected/808b9b44-8c2d-4e08-a110-e6660181bea3-kube-api-access-chwcp\") pod \"authorino-557f876f69-8jrhr\" (UID: \"808b9b44-8c2d-4e08-a110-e6660181bea3\") " pod="kuadrant-system/authorino-557f876f69-8jrhr"
Sep 10 15:50:44.129003 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.128969    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5-tls-cert\") pod \"authorino-6d4dfc649f-q524n\" (UID: \"e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5\") " pod="kuadrant-system/authorino-6d4dfc649f-q524n"
Sep 10 15:50:44.129190 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.129025    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chwcp\" (UniqueName: \"kubernetes.io/projected/808b9b44-8c2d-4e08-a110-e6660181bea3-kube-api-access-chwcp\") pod \"authorino-557f876f69-8jrhr\" (UID: \"808b9b44-8c2d-4e08-a110-e6660181bea3\") " pod="kuadrant-system/authorino-557f876f69-8jrhr"
Sep 10 15:50:44.129190 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.129051    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqwkz\" (UniqueName: \"kubernetes.io/projected/e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5-kube-api-access-cqwkz\") pod \"authorino-6d4dfc649f-q524n\" (UID: \"e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5\") " pod="kuadrant-system/authorino-6d4dfc649f-q524n"
Sep 10 15:50:44.144182 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.144146    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chwcp\" (UniqueName: \"kubernetes.io/projected/808b9b44-8c2d-4e08-a110-e6660181bea3-kube-api-access-chwcp\") pod \"authorino-557f876f69-8jrhr\" (UID: \"808b9b44-8c2d-4e08-a110-e6660181bea3\") " pod="kuadrant-system/authorino-557f876f69-8jrhr"
Sep 10 15:50:44.157961 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.157926    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-q524n"]
Sep 10 15:50:44.158184 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:50:44.158165    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cqwkz tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-q524n" podUID="e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5"
Sep 10 15:50:44.229835 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.229745    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqwkz\" (UniqueName: \"kubernetes.io/projected/e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5-kube-api-access-cqwkz\") pod \"authorino-6d4dfc649f-q524n\" (UID: \"e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5\") " pod="kuadrant-system/authorino-6d4dfc649f-q524n"
Sep 10 15:50:44.229835 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.229807    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5-tls-cert\") pod \"authorino-6d4dfc649f-q524n\" (UID: \"e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5\") " pod="kuadrant-system/authorino-6d4dfc649f-q524n"
Sep 10 15:50:44.232223 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.232201    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5-tls-cert\") pod \"authorino-6d4dfc649f-q524n\" (UID: \"e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5\") " pod="kuadrant-system/authorino-6d4dfc649f-q524n"
Sep 10 15:50:44.238489 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.238461    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-85654dd45c-fgfcw"]
Sep 10 15:50:44.241901 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.241883    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85654dd45c-fgfcw"
Sep 10 15:50:44.250438 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.250417    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85654dd45c-fgfcw"]
Sep 10 15:50:44.257135 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.257106    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqwkz\" (UniqueName: \"kubernetes.io/projected/e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5-kube-api-access-cqwkz\") pod \"authorino-6d4dfc649f-q524n\" (UID: \"e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5\") " pod="kuadrant-system/authorino-6d4dfc649f-q524n"
Sep 10 15:50:44.331055 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.331016    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44wj9\" (UniqueName: \"kubernetes.io/projected/8cb59db5-e927-4c98-a824-256619e14442-kube-api-access-44wj9\") pod \"authorino-85654dd45c-fgfcw\" (UID: \"8cb59db5-e927-4c98-a824-256619e14442\") " pod="kuadrant-system/authorino-85654dd45c-fgfcw"
Sep 10 15:50:44.331234 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.331127    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8cb59db5-e927-4c98-a824-256619e14442-tls-cert\") pod \"authorino-85654dd45c-fgfcw\" (UID: \"8cb59db5-e927-4c98-a824-256619e14442\") " pod="kuadrant-system/authorino-85654dd45c-fgfcw"
Sep 10 15:50:44.360433 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.360403    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-q524n"
Sep 10 15:50:44.360599 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.360403    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-8jrhr"
Sep 10 15:50:44.367816 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.367786    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-q524n"
Sep 10 15:50:44.370681 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.370661    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-8jrhr"
Sep 10 15:50:44.432213 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.432177    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5-tls-cert\") pod \"e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5\" (UID: \"e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5\") "
Sep 10 15:50:44.432425 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.432256    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chwcp\" (UniqueName: \"kubernetes.io/projected/808b9b44-8c2d-4e08-a110-e6660181bea3-kube-api-access-chwcp\") pod \"808b9b44-8c2d-4e08-a110-e6660181bea3\" (UID: \"808b9b44-8c2d-4e08-a110-e6660181bea3\") "
Sep 10 15:50:44.432425 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.432289    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqwkz\" (UniqueName: \"kubernetes.io/projected/e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5-kube-api-access-cqwkz\") pod \"e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5\" (UID: \"e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5\") "
Sep 10 15:50:44.432425 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.432371    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8cb59db5-e927-4c98-a824-256619e14442-tls-cert\") pod \"authorino-85654dd45c-fgfcw\" (UID: \"8cb59db5-e927-4c98-a824-256619e14442\") " pod="kuadrant-system/authorino-85654dd45c-fgfcw"
Sep 10 15:50:44.432602 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.432426    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44wj9\" (UniqueName: \"kubernetes.io/projected/8cb59db5-e927-4c98-a824-256619e14442-kube-api-access-44wj9\") pod \"authorino-85654dd45c-fgfcw\" (UID: \"8cb59db5-e927-4c98-a824-256619e14442\") " pod="kuadrant-system/authorino-85654dd45c-fgfcw"
Sep 10 15:50:44.434647 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.434615    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808b9b44-8c2d-4e08-a110-e6660181bea3-kube-api-access-chwcp" (OuterVolumeSpecName: "kube-api-access-chwcp") pod "808b9b44-8c2d-4e08-a110-e6660181bea3" (UID: "808b9b44-8c2d-4e08-a110-e6660181bea3"). InnerVolumeSpecName "kube-api-access-chwcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:44.434862 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.434842    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5-kube-api-access-cqwkz" (OuterVolumeSpecName: "kube-api-access-cqwkz") pod "e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5" (UID: "e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5"). InnerVolumeSpecName "kube-api-access-cqwkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:44.434911 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.434864    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5" (UID: "e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:50:44.434961 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.434942    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8cb59db5-e927-4c98-a824-256619e14442-tls-cert\") pod \"authorino-85654dd45c-fgfcw\" (UID: \"8cb59db5-e927-4c98-a824-256619e14442\") " pod="kuadrant-system/authorino-85654dd45c-fgfcw"
Sep 10 15:50:44.440410 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.440382    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44wj9\" (UniqueName: \"kubernetes.io/projected/8cb59db5-e927-4c98-a824-256619e14442-kube-api-access-44wj9\") pod \"authorino-85654dd45c-fgfcw\" (UID: \"8cb59db5-e927-4c98-a824-256619e14442\") " pod="kuadrant-system/authorino-85654dd45c-fgfcw"
Sep 10 15:50:44.533537 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.533504    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chwcp\" (UniqueName: \"kubernetes.io/projected/808b9b44-8c2d-4e08-a110-e6660181bea3-kube-api-access-chwcp\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:50:44.533537 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.533533    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqwkz\" (UniqueName: \"kubernetes.io/projected/e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5-kube-api-access-cqwkz\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:50:44.533537 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.533543    2572 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5-tls-cert\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:50:44.551408 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.551382    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85654dd45c-fgfcw"
Sep 10 15:50:44.819925 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:44.819897    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85654dd45c-fgfcw"]
Sep 10 15:50:44.822207 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:50:44.822179    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cb59db5_e927_4c98_a824_256619e14442.slice/crio-c51eea607170bebcc7e11512236f3d373d6be620eb3c3707593ebd7f20754350 WatchSource:0}: Error finding container c51eea607170bebcc7e11512236f3d373d6be620eb3c3707593ebd7f20754350: Status 404 returned error can't find the container with id c51eea607170bebcc7e11512236f3d373d6be620eb3c3707593ebd7f20754350
Sep 10 15:50:45.367933 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:45.367893    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85654dd45c-fgfcw" event={"ID":"8cb59db5-e927-4c98-a824-256619e14442","Type":"ContainerStarted","Data":"699fce0fe5d3eb6166f671c75ae0e21ff4c52f48c57038c3df414ab8fb390e99"}
Sep 10 15:50:45.367933 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:45.367914    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-q524n"
Sep 10 15:50:45.368456 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:45.367942    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85654dd45c-fgfcw" event={"ID":"8cb59db5-e927-4c98-a824-256619e14442","Type":"ContainerStarted","Data":"c51eea607170bebcc7e11512236f3d373d6be620eb3c3707593ebd7f20754350"}
Sep 10 15:50:45.368456 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:45.367907    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-8jrhr"
Sep 10 15:50:45.387434 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:45.387372    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-85654dd45c-fgfcw" podStartSLOduration=1.05192055 podStartE2EDuration="1.387351084s" totalImagesPullingTime="335.430534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:45.387094747 +0000 UTC m=+518.059504831" watchObservedRunningTime="2026-09-10 15:50:45.387351084 +0000 UTC m=+518.059761167"
Sep 10 15:50:45.409247 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:45.409145    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-8jrhr"]
Sep 10 15:50:45.416627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:45.415959    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cc5js"]
Sep 10 15:50:45.416627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:45.416248    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-cc5js" podUID="8683732e-50a1-46ee-96b3-3128a74dc812" containerName="authorino" containerID="cri-o://3717ab938047bada1772872686dd5b3221347f8af41678d75c040209d62ea31b" gracePeriod=30
Sep 10 15:50:45.419344 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:45.419284    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-8jrhr"]
Sep 10 15:50:45.451115 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:45.451078    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-q524n"]
Sep 10 15:50:45.459095 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:45.459065    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-q524n"]
Sep 10 15:50:45.654118 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:45.654094    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cc5js"
Sep 10 15:50:45.744164 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:45.744063    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vh8tr\" (UniqueName: \"kubernetes.io/projected/8683732e-50a1-46ee-96b3-3128a74dc812-kube-api-access-vh8tr\") pod \"8683732e-50a1-46ee-96b3-3128a74dc812\" (UID: \"8683732e-50a1-46ee-96b3-3128a74dc812\") "
Sep 10 15:50:45.746189 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:45.746160    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8683732e-50a1-46ee-96b3-3128a74dc812-kube-api-access-vh8tr" (OuterVolumeSpecName: "kube-api-access-vh8tr") pod "8683732e-50a1-46ee-96b3-3128a74dc812" (UID: "8683732e-50a1-46ee-96b3-3128a74dc812"). InnerVolumeSpecName "kube-api-access-vh8tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:45.845349 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:45.845260    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh8tr\" (UniqueName: \"kubernetes.io/projected/8683732e-50a1-46ee-96b3-3128a74dc812-kube-api-access-vh8tr\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:50:45.993370 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:45.993331    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="808b9b44-8c2d-4e08-a110-e6660181bea3" path="/var/lib/kubelet/pods/808b9b44-8c2d-4e08-a110-e6660181bea3/volumes"
Sep 10 15:50:45.993672 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:45.993654    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5" path="/var/lib/kubelet/pods/e8738312-2fa7-4f3a-a89d-ce14ec3a4ea5/volumes"
Sep 10 15:50:46.373710 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:46.373674    2572 generic.go:356] "Generic (PLEG): container finished" podID="8683732e-50a1-46ee-96b3-3128a74dc812" containerID="3717ab938047bada1772872686dd5b3221347f8af41678d75c040209d62ea31b" exitCode=0
Sep 10 15:50:46.374236 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:46.373720    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cc5js"
Sep 10 15:50:46.374236 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:46.373759    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cc5js" event={"ID":"8683732e-50a1-46ee-96b3-3128a74dc812","Type":"ContainerDied","Data":"3717ab938047bada1772872686dd5b3221347f8af41678d75c040209d62ea31b"}
Sep 10 15:50:46.374236 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:46.373800    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cc5js" event={"ID":"8683732e-50a1-46ee-96b3-3128a74dc812","Type":"ContainerDied","Data":"30096cb6f078833523c6aa45370df62377ecbc7bb70d69dd9ee2c612bd77e699"}
Sep 10 15:50:46.374236 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:46.373821    2572 scope.go:117] "RemoveContainer" containerID="3717ab938047bada1772872686dd5b3221347f8af41678d75c040209d62ea31b"
Sep 10 15:50:46.382246 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:46.382228    2572 scope.go:117] "RemoveContainer" containerID="3717ab938047bada1772872686dd5b3221347f8af41678d75c040209d62ea31b"
Sep 10 15:50:46.382576 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:50:46.382557    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3717ab938047bada1772872686dd5b3221347f8af41678d75c040209d62ea31b\": container with ID starting with 3717ab938047bada1772872686dd5b3221347f8af41678d75c040209d62ea31b not found: ID does not exist" containerID="3717ab938047bada1772872686dd5b3221347f8af41678d75c040209d62ea31b"
Sep 10 15:50:46.382631 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:46.382586    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3717ab938047bada1772872686dd5b3221347f8af41678d75c040209d62ea31b"} err="failed to get container status \"3717ab938047bada1772872686dd5b3221347f8af41678d75c040209d62ea31b\": rpc error: code = NotFound desc = could not find container \"3717ab938047bada1772872686dd5b3221347f8af41678d75c040209d62ea31b\": container with ID starting with 3717ab938047bada1772872686dd5b3221347f8af41678d75c040209d62ea31b not found: ID does not exist"
Sep 10 15:50:46.389549 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:46.389517    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cc5js"]
Sep 10 15:50:46.393913 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:46.393886    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cc5js"]
Sep 10 15:50:47.992287 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:50:47.992246    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8683732e-50a1-46ee-96b3-3128a74dc812" path="/var/lib/kubelet/pods/8683732e-50a1-46ee-96b3-3128a74dc812/volumes"
Sep 10 15:52:01.741347 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.741280    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"]
Sep 10 15:52:01.741781 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.741651    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8683732e-50a1-46ee-96b3-3128a74dc812" containerName="authorino"
Sep 10 15:52:01.741781 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.741664    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8683732e-50a1-46ee-96b3-3128a74dc812" containerName="authorino"
Sep 10 15:52:01.741781 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.741737    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8683732e-50a1-46ee-96b3-3128a74dc812" containerName="authorino"
Sep 10 15:52:01.744734 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.744715    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:01.746943 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.746918    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 15:52:01.747080 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.746994    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4b9fq\""
Sep 10 15:52:01.747649 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.747621    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 15:52:01.747727 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.747632    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 10 15:52:01.756983 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.756947    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"]
Sep 10 15:52:01.796650 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.796608    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29aa605d-c128-4458-9dc9-a7c0733a4855-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4p8lp\" (UID: \"29aa605d-c128-4458-9dc9-a7c0733a4855\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:01.796836 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.796669    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29aa605d-c128-4458-9dc9-a7c0733a4855-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4p8lp\" (UID: \"29aa605d-c128-4458-9dc9-a7c0733a4855\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:01.796836 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.796760    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29aa605d-c128-4458-9dc9-a7c0733a4855-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4p8lp\" (UID: \"29aa605d-c128-4458-9dc9-a7c0733a4855\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:01.796836 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.796791    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29aa605d-c128-4458-9dc9-a7c0733a4855-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4p8lp\" (UID: \"29aa605d-c128-4458-9dc9-a7c0733a4855\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:01.796836 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.796810    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29aa605d-c128-4458-9dc9-a7c0733a4855-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4p8lp\" (UID: \"29aa605d-c128-4458-9dc9-a7c0733a4855\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:01.796836 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.796835    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptzxk\" (UniqueName: \"kubernetes.io/projected/29aa605d-c128-4458-9dc9-a7c0733a4855-kube-api-access-ptzxk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4p8lp\" (UID: \"29aa605d-c128-4458-9dc9-a7c0733a4855\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:01.897652 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.897604    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29aa605d-c128-4458-9dc9-a7c0733a4855-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4p8lp\" (UID: \"29aa605d-c128-4458-9dc9-a7c0733a4855\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:01.897850 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.897665    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29aa605d-c128-4458-9dc9-a7c0733a4855-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4p8lp\" (UID: \"29aa605d-c128-4458-9dc9-a7c0733a4855\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:01.897850 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.897681    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29aa605d-c128-4458-9dc9-a7c0733a4855-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4p8lp\" (UID: \"29aa605d-c128-4458-9dc9-a7c0733a4855\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:01.897850 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.897698    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29aa605d-c128-4458-9dc9-a7c0733a4855-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4p8lp\" (UID: \"29aa605d-c128-4458-9dc9-a7c0733a4855\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:01.897850 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.897722    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptzxk\" (UniqueName: \"kubernetes.io/projected/29aa605d-c128-4458-9dc9-a7c0733a4855-kube-api-access-ptzxk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4p8lp\" (UID: \"29aa605d-c128-4458-9dc9-a7c0733a4855\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:01.897850 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.897772    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29aa605d-c128-4458-9dc9-a7c0733a4855-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4p8lp\" (UID: \"29aa605d-c128-4458-9dc9-a7c0733a4855\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:01.898121 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.898093    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29aa605d-c128-4458-9dc9-a7c0733a4855-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4p8lp\" (UID: \"29aa605d-c128-4458-9dc9-a7c0733a4855\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:01.898172 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.898111    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29aa605d-c128-4458-9dc9-a7c0733a4855-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4p8lp\" (UID: \"29aa605d-c128-4458-9dc9-a7c0733a4855\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:01.898172 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.898158    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29aa605d-c128-4458-9dc9-a7c0733a4855-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4p8lp\" (UID: \"29aa605d-c128-4458-9dc9-a7c0733a4855\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:01.900134 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.900109    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29aa605d-c128-4458-9dc9-a7c0733a4855-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4p8lp\" (UID: \"29aa605d-c128-4458-9dc9-a7c0733a4855\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:01.900293 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.900275    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29aa605d-c128-4458-9dc9-a7c0733a4855-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4p8lp\" (UID: \"29aa605d-c128-4458-9dc9-a7c0733a4855\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:01.905230 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:01.905198    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptzxk\" (UniqueName: \"kubernetes.io/projected/29aa605d-c128-4458-9dc9-a7c0733a4855-kube-api-access-ptzxk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4p8lp\" (UID: \"29aa605d-c128-4458-9dc9-a7c0733a4855\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:02.057040 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:02.056995    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:02.329380 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:02.329350    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"]
Sep 10 15:52:02.331457 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:52:02.331422    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29aa605d_c128_4458_9dc9_a7c0733a4855.slice/crio-bb70e83806da99dfc44143edc435e98d3906439a3beab317411ec8b0bc0d834c WatchSource:0}: Error finding container bb70e83806da99dfc44143edc435e98d3906439a3beab317411ec8b0bc0d834c: Status 404 returned error can't find the container with id bb70e83806da99dfc44143edc435e98d3906439a3beab317411ec8b0bc0d834c
Sep 10 15:52:02.772403 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:02.772293    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp" event={"ID":"29aa605d-c128-4458-9dc9-a7c0733a4855","Type":"ContainerStarted","Data":"bb70e83806da99dfc44143edc435e98d3906439a3beab317411ec8b0bc0d834c"}
Sep 10 15:52:03.779440 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:03.779342    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp" event={"ID":"29aa605d-c128-4458-9dc9-a7c0733a4855","Type":"ContainerStarted","Data":"e3bf1fb297fbd6c1dbf0d3065670fbaa5dd77921d0c480d5e425c25a386ac367"}
Sep 10 15:52:03.799416 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:03.799351    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp" podStartSLOduration=1.7218831460000001 podStartE2EDuration="2.799332007s" totalImagesPullingTime="1.077448861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:52:03.796847317 +0000 UTC m=+596.469257401" watchObservedRunningTime="2026-09-10 15:52:03.799332007 +0000 UTC m=+596.471742084"
Sep 10 15:52:07.791954 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:07.791922    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7sm72_8ddd4dc1-1122-41e0-a54a-67fb8008d7b9/ovn-acl-logging/0.log"
Sep 10 15:52:07.795957 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:07.795934    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7sm72_8ddd4dc1-1122-41e0-a54a-67fb8008d7b9/ovn-acl-logging/0.log"
Sep 10 15:52:12.057529 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:12.057483    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:12.057991 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:12.057547    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:12.070310 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:12.070262    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:12.834778 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:12.834664    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4p8lp"
Sep 10 15:52:28.655365 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.655328    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"]
Sep 10 15:52:28.660111 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.660087    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:28.662627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.662602    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 15:52:28.667171 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.667143    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"]
Sep 10 15:52:28.754211 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.754172    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a722d5-5379-4a20-9980-f491822d582c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7\" (UID: \"62a722d5-5379-4a20-9980-f491822d582c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:28.754473 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.754229    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a722d5-5379-4a20-9980-f491822d582c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7\" (UID: \"62a722d5-5379-4a20-9980-f491822d582c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:28.754473 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.754262    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a722d5-5379-4a20-9980-f491822d582c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7\" (UID: \"62a722d5-5379-4a20-9980-f491822d582c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:28.754473 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.754342    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a722d5-5379-4a20-9980-f491822d582c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7\" (UID: \"62a722d5-5379-4a20-9980-f491822d582c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:28.754473 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.754363    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8qgp\" (UniqueName: \"kubernetes.io/projected/62a722d5-5379-4a20-9980-f491822d582c-kube-api-access-x8qgp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7\" (UID: \"62a722d5-5379-4a20-9980-f491822d582c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:28.754473 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.754398    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a722d5-5379-4a20-9980-f491822d582c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7\" (UID: \"62a722d5-5379-4a20-9980-f491822d582c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:28.855513 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.855471    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a722d5-5379-4a20-9980-f491822d582c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7\" (UID: \"62a722d5-5379-4a20-9980-f491822d582c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:28.855513 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.855516    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8qgp\" (UniqueName: \"kubernetes.io/projected/62a722d5-5379-4a20-9980-f491822d582c-kube-api-access-x8qgp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7\" (UID: \"62a722d5-5379-4a20-9980-f491822d582c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:28.855755 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.855557    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a722d5-5379-4a20-9980-f491822d582c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7\" (UID: \"62a722d5-5379-4a20-9980-f491822d582c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:28.855755 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.855586    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a722d5-5379-4a20-9980-f491822d582c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7\" (UID: \"62a722d5-5379-4a20-9980-f491822d582c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:28.855755 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.855621    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a722d5-5379-4a20-9980-f491822d582c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7\" (UID: \"62a722d5-5379-4a20-9980-f491822d582c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:28.855755 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.855657    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a722d5-5379-4a20-9980-f491822d582c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7\" (UID: \"62a722d5-5379-4a20-9980-f491822d582c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:28.856093 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.856061    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a722d5-5379-4a20-9980-f491822d582c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7\" (UID: \"62a722d5-5379-4a20-9980-f491822d582c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:28.856221 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.856087    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a722d5-5379-4a20-9980-f491822d582c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7\" (UID: \"62a722d5-5379-4a20-9980-f491822d582c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:28.856221 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.856155    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a722d5-5379-4a20-9980-f491822d582c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7\" (UID: \"62a722d5-5379-4a20-9980-f491822d582c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:28.857976 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.857952    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a722d5-5379-4a20-9980-f491822d582c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7\" (UID: \"62a722d5-5379-4a20-9980-f491822d582c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:28.858150 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.858132    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a722d5-5379-4a20-9980-f491822d582c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7\" (UID: \"62a722d5-5379-4a20-9980-f491822d582c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:28.864542 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.864518    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8qgp\" (UniqueName: \"kubernetes.io/projected/62a722d5-5379-4a20-9980-f491822d582c-kube-api-access-x8qgp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7\" (UID: \"62a722d5-5379-4a20-9980-f491822d582c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:28.971439 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:28.971342    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:29.246132 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:29.246102    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"]
Sep 10 15:52:29.247731 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:52:29.247703    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a722d5_5379_4a20_9980_f491822d582c.slice/crio-83cfaf9a883346952208492f12cc2c47a4da645098b0237270ed350075dcc3f3 WatchSource:0}: Error finding container 83cfaf9a883346952208492f12cc2c47a4da645098b0237270ed350075dcc3f3: Status 404 returned error can't find the container with id 83cfaf9a883346952208492f12cc2c47a4da645098b0237270ed350075dcc3f3
Sep 10 15:52:29.249060 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:29.249039    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:52:29.910371 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:29.910331    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7" event={"ID":"62a722d5-5379-4a20-9980-f491822d582c","Type":"ContainerStarted","Data":"2c4a9281bf8875465e4b76901f5df8a818e09ef0fb3d21f011bfc68e946cac3c"}
Sep 10 15:52:29.910778 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:29.910415    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7" event={"ID":"62a722d5-5379-4a20-9980-f491822d582c","Type":"ContainerStarted","Data":"83cfaf9a883346952208492f12cc2c47a4da645098b0237270ed350075dcc3f3"}
Sep 10 15:52:29.929510 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:29.929454    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7" podStartSLOduration=1.708820357 podStartE2EDuration="1.929437199s" totalImagesPullingTime="220.616842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:52:29.926867611 +0000 UTC m=+622.599277695" watchObservedRunningTime="2026-09-10 15:52:29.929437199 +0000 UTC m=+622.601847282"
Sep 10 15:52:38.972382 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:38.972339    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:38.972880 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:38.972417    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:38.984806 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:38.984772    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:52:39.979438 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:52:39.979401    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7"
Sep 10 15:53:22.069115 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.069074    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"]
Sep 10 15:53:22.072478 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.072458    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.074688 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.074668    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 15:53:22.081675 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.081650    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"]
Sep 10 15:53:22.224173 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.224134    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebc04cfb-6032-4bb9-9ec4-5146e22719d8-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz\" (UID: \"ebc04cfb-6032-4bb9-9ec4-5146e22719d8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.224373 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.224186    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebc04cfb-6032-4bb9-9ec4-5146e22719d8-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz\" (UID: \"ebc04cfb-6032-4bb9-9ec4-5146e22719d8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.224373 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.224271    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebc04cfb-6032-4bb9-9ec4-5146e22719d8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz\" (UID: \"ebc04cfb-6032-4bb9-9ec4-5146e22719d8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.224373 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.224322    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc04cfb-6032-4bb9-9ec4-5146e22719d8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz\" (UID: \"ebc04cfb-6032-4bb9-9ec4-5146e22719d8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.224488 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.224367    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbnmt\" (UniqueName: \"kubernetes.io/projected/ebc04cfb-6032-4bb9-9ec4-5146e22719d8-kube-api-access-hbnmt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz\" (UID: \"ebc04cfb-6032-4bb9-9ec4-5146e22719d8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.224488 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.224403    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc04cfb-6032-4bb9-9ec4-5146e22719d8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz\" (UID: \"ebc04cfb-6032-4bb9-9ec4-5146e22719d8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.325324 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.325226    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebc04cfb-6032-4bb9-9ec4-5146e22719d8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz\" (UID: \"ebc04cfb-6032-4bb9-9ec4-5146e22719d8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.325324 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.325263    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc04cfb-6032-4bb9-9ec4-5146e22719d8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz\" (UID: \"ebc04cfb-6032-4bb9-9ec4-5146e22719d8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.325324 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.325281    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbnmt\" (UniqueName: \"kubernetes.io/projected/ebc04cfb-6032-4bb9-9ec4-5146e22719d8-kube-api-access-hbnmt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz\" (UID: \"ebc04cfb-6032-4bb9-9ec4-5146e22719d8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.325324 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.325325    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc04cfb-6032-4bb9-9ec4-5146e22719d8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz\" (UID: \"ebc04cfb-6032-4bb9-9ec4-5146e22719d8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.325662 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.325376    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebc04cfb-6032-4bb9-9ec4-5146e22719d8-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz\" (UID: \"ebc04cfb-6032-4bb9-9ec4-5146e22719d8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.325662 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.325403    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebc04cfb-6032-4bb9-9ec4-5146e22719d8-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz\" (UID: \"ebc04cfb-6032-4bb9-9ec4-5146e22719d8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.325743 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.325709    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebc04cfb-6032-4bb9-9ec4-5146e22719d8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz\" (UID: \"ebc04cfb-6032-4bb9-9ec4-5146e22719d8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.325804 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.325755    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc04cfb-6032-4bb9-9ec4-5146e22719d8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz\" (UID: \"ebc04cfb-6032-4bb9-9ec4-5146e22719d8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.325804 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.325780    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebc04cfb-6032-4bb9-9ec4-5146e22719d8-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz\" (UID: \"ebc04cfb-6032-4bb9-9ec4-5146e22719d8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.327567 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.327546    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebc04cfb-6032-4bb9-9ec4-5146e22719d8-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz\" (UID: \"ebc04cfb-6032-4bb9-9ec4-5146e22719d8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.327814 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.327795    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc04cfb-6032-4bb9-9ec4-5146e22719d8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz\" (UID: \"ebc04cfb-6032-4bb9-9ec4-5146e22719d8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.332993 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.332964    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbnmt\" (UniqueName: \"kubernetes.io/projected/ebc04cfb-6032-4bb9-9ec4-5146e22719d8-kube-api-access-hbnmt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz\" (UID: \"ebc04cfb-6032-4bb9-9ec4-5146e22719d8\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.381956 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.381915    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:22.659201 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:22.659175    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"]
Sep 10 15:53:22.661093 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:53:22.661067    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebc04cfb_6032_4bb9_9ec4_5146e22719d8.slice/crio-dd28ef2d87328de3d83bf644911c52e43a7bc8dabbd5b65d5fe27ab83413c1de WatchSource:0}: Error finding container dd28ef2d87328de3d83bf644911c52e43a7bc8dabbd5b65d5fe27ab83413c1de: Status 404 returned error can't find the container with id dd28ef2d87328de3d83bf644911c52e43a7bc8dabbd5b65d5fe27ab83413c1de
Sep 10 15:53:23.194059 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:23.194018    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz" event={"ID":"ebc04cfb-6032-4bb9-9ec4-5146e22719d8","Type":"ContainerStarted","Data":"38aabcf319eec36ca169340e09d5eeae7861d657133df2a6619605388ddcf596"}
Sep 10 15:53:23.194487 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:23.194072    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz" event={"ID":"ebc04cfb-6032-4bb9-9ec4-5146e22719d8","Type":"ContainerStarted","Data":"dd28ef2d87328de3d83bf644911c52e43a7bc8dabbd5b65d5fe27ab83413c1de"}
Sep 10 15:53:23.212396 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:23.212346    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz" podStartSLOduration=1.017596475 podStartE2EDuration="1.212331158s" totalImagesPullingTime="194.734683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:53:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:53:23.21124543 +0000 UTC m=+675.883655514" watchObservedRunningTime="2026-09-10 15:53:23.212331158 +0000 UTC m=+675.884741241"
Sep 10 15:53:32.382558 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:32.382518    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:32.382929 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:32.382574    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:32.394965 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:32.394944    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:53:33.254792 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:53:33.254764    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz"
Sep 10 15:54:44.315116 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:44.315075    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-565c6b8cd8-p694g"]
Sep 10 15:54:44.318492 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:44.318470    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-565c6b8cd8-p694g"
Sep 10 15:54:44.326780 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:44.326750    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-565c6b8cd8-p694g"]
Sep 10 15:54:44.435762 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:44.435723    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8snzs\" (UniqueName: \"kubernetes.io/projected/f0099fd3-4549-4e2a-ac0d-9a14da2b3b35-kube-api-access-8snzs\") pod \"authorino-565c6b8cd8-p694g\" (UID: \"f0099fd3-4549-4e2a-ac0d-9a14da2b3b35\") " pod="kuadrant-system/authorino-565c6b8cd8-p694g"
Sep 10 15:54:44.435962 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:44.435787    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0099fd3-4549-4e2a-ac0d-9a14da2b3b35-tls-cert\") pod \"authorino-565c6b8cd8-p694g\" (UID: \"f0099fd3-4549-4e2a-ac0d-9a14da2b3b35\") " pod="kuadrant-system/authorino-565c6b8cd8-p694g"
Sep 10 15:54:44.536897 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:44.536856    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8snzs\" (UniqueName: \"kubernetes.io/projected/f0099fd3-4549-4e2a-ac0d-9a14da2b3b35-kube-api-access-8snzs\") pod \"authorino-565c6b8cd8-p694g\" (UID: \"f0099fd3-4549-4e2a-ac0d-9a14da2b3b35\") " pod="kuadrant-system/authorino-565c6b8cd8-p694g"
Sep 10 15:54:44.537093 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:44.536913    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0099fd3-4549-4e2a-ac0d-9a14da2b3b35-tls-cert\") pod \"authorino-565c6b8cd8-p694g\" (UID: \"f0099fd3-4549-4e2a-ac0d-9a14da2b3b35\") " pod="kuadrant-system/authorino-565c6b8cd8-p694g"
Sep 10 15:54:44.539434 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:44.539399    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0099fd3-4549-4e2a-ac0d-9a14da2b3b35-tls-cert\") pod \"authorino-565c6b8cd8-p694g\" (UID: \"f0099fd3-4549-4e2a-ac0d-9a14da2b3b35\") " pod="kuadrant-system/authorino-565c6b8cd8-p694g"
Sep 10 15:54:44.544893 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:44.544863    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8snzs\" (UniqueName: \"kubernetes.io/projected/f0099fd3-4549-4e2a-ac0d-9a14da2b3b35-kube-api-access-8snzs\") pod \"authorino-565c6b8cd8-p694g\" (UID: \"f0099fd3-4549-4e2a-ac0d-9a14da2b3b35\") " pod="kuadrant-system/authorino-565c6b8cd8-p694g"
Sep 10 15:54:44.628187 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:44.628099    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-565c6b8cd8-p694g"
Sep 10 15:54:44.902988 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:44.902954    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-565c6b8cd8-p694g"]
Sep 10 15:54:44.904876 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:54:44.904843    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0099fd3_4549_4e2a_ac0d_9a14da2b3b35.slice/crio-34575b01e986c59e8d5ee084196b1b42f8900398d0ebc938f5d998238b1230f3 WatchSource:0}: Error finding container 34575b01e986c59e8d5ee084196b1b42f8900398d0ebc938f5d998238b1230f3: Status 404 returned error can't find the container with id 34575b01e986c59e8d5ee084196b1b42f8900398d0ebc938f5d998238b1230f3
Sep 10 15:54:45.637124 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:45.637091    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-565c6b8cd8-p694g" event={"ID":"f0099fd3-4549-4e2a-ac0d-9a14da2b3b35","Type":"ContainerStarted","Data":"d3fe86b19346edafc296c3508876c8c85f180a5b1535d6ac4521505d7d16932b"}
Sep 10 15:54:45.637504 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:45.637133    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-565c6b8cd8-p694g" event={"ID":"f0099fd3-4549-4e2a-ac0d-9a14da2b3b35","Type":"ContainerStarted","Data":"34575b01e986c59e8d5ee084196b1b42f8900398d0ebc938f5d998238b1230f3"}
Sep 10 15:54:45.654748 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:45.654685    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-565c6b8cd8-p694g" podStartSLOduration=1.036599392 podStartE2EDuration="1.65466196s" totalImagesPullingTime="618.062568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:45.65194254 +0000 UTC m=+758.324352636" watchObservedRunningTime="2026-09-10 15:54:45.65466196 +0000 UTC m=+758.327072048"
Sep 10 15:54:45.678251 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:45.677915    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85654dd45c-fgfcw"]
Sep 10 15:54:45.678251 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:45.678154    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-85654dd45c-fgfcw" podUID="8cb59db5-e927-4c98-a824-256619e14442" containerName="authorino" containerID="cri-o://699fce0fe5d3eb6166f671c75ae0e21ff4c52f48c57038c3df414ab8fb390e99" gracePeriod=30
Sep 10 15:54:45.941563 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:45.941533    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85654dd45c-fgfcw"
Sep 10 15:54:46.050514 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:46.050482    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8cb59db5-e927-4c98-a824-256619e14442-tls-cert\") pod \"8cb59db5-e927-4c98-a824-256619e14442\" (UID: \"8cb59db5-e927-4c98-a824-256619e14442\") "
Sep 10 15:54:46.050692 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:46.050559    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44wj9\" (UniqueName: \"kubernetes.io/projected/8cb59db5-e927-4c98-a824-256619e14442-kube-api-access-44wj9\") pod \"8cb59db5-e927-4c98-a824-256619e14442\" (UID: \"8cb59db5-e927-4c98-a824-256619e14442\") "
Sep 10 15:54:46.052713 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:46.052683    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb59db5-e927-4c98-a824-256619e14442-kube-api-access-44wj9" (OuterVolumeSpecName: "kube-api-access-44wj9") pod "8cb59db5-e927-4c98-a824-256619e14442" (UID: "8cb59db5-e927-4c98-a824-256619e14442"). InnerVolumeSpecName "kube-api-access-44wj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:54:46.062098 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:46.062063    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb59db5-e927-4c98-a824-256619e14442-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "8cb59db5-e927-4c98-a824-256619e14442" (UID: "8cb59db5-e927-4c98-a824-256619e14442"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:54:46.151282 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:46.151250    2572 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8cb59db5-e927-4c98-a824-256619e14442-tls-cert\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:54:46.151282 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:46.151276    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44wj9\" (UniqueName: \"kubernetes.io/projected/8cb59db5-e927-4c98-a824-256619e14442-kube-api-access-44wj9\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:54:46.643809 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:46.643773    2572 generic.go:356] "Generic (PLEG): container finished" podID="8cb59db5-e927-4c98-a824-256619e14442" containerID="699fce0fe5d3eb6166f671c75ae0e21ff4c52f48c57038c3df414ab8fb390e99" exitCode=0
Sep 10 15:54:46.644284 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:46.643826    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85654dd45c-fgfcw"
Sep 10 15:54:46.644284 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:46.643861    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85654dd45c-fgfcw" event={"ID":"8cb59db5-e927-4c98-a824-256619e14442","Type":"ContainerDied","Data":"699fce0fe5d3eb6166f671c75ae0e21ff4c52f48c57038c3df414ab8fb390e99"}
Sep 10 15:54:46.644284 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:46.643903    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85654dd45c-fgfcw" event={"ID":"8cb59db5-e927-4c98-a824-256619e14442","Type":"ContainerDied","Data":"c51eea607170bebcc7e11512236f3d373d6be620eb3c3707593ebd7f20754350"}
Sep 10 15:54:46.644284 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:46.643921    2572 scope.go:117] "RemoveContainer" containerID="699fce0fe5d3eb6166f671c75ae0e21ff4c52f48c57038c3df414ab8fb390e99"
Sep 10 15:54:46.653840 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:46.653821    2572 scope.go:117] "RemoveContainer" containerID="699fce0fe5d3eb6166f671c75ae0e21ff4c52f48c57038c3df414ab8fb390e99"
Sep 10 15:54:46.654216 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:54:46.654196    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"699fce0fe5d3eb6166f671c75ae0e21ff4c52f48c57038c3df414ab8fb390e99\": container with ID starting with 699fce0fe5d3eb6166f671c75ae0e21ff4c52f48c57038c3df414ab8fb390e99 not found: ID does not exist" containerID="699fce0fe5d3eb6166f671c75ae0e21ff4c52f48c57038c3df414ab8fb390e99"
Sep 10 15:54:46.654320 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:46.654224    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"699fce0fe5d3eb6166f671c75ae0e21ff4c52f48c57038c3df414ab8fb390e99"} err="failed to get container status \"699fce0fe5d3eb6166f671c75ae0e21ff4c52f48c57038c3df414ab8fb390e99\": rpc error: code = NotFound desc = could not find container \"699fce0fe5d3eb6166f671c75ae0e21ff4c52f48c57038c3df414ab8fb390e99\": container with ID starting with 699fce0fe5d3eb6166f671c75ae0e21ff4c52f48c57038c3df414ab8fb390e99 not found: ID does not exist"
Sep 10 15:54:46.667851 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:46.667810    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85654dd45c-fgfcw"]
Sep 10 15:54:46.671676 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:46.671647    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-85654dd45c-fgfcw"]
Sep 10 15:54:47.992494 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:54:47.992458    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cb59db5-e927-4c98-a824-256619e14442" path="/var/lib/kubelet/pods/8cb59db5-e927-4c98-a824-256619e14442/volumes"
Sep 10 15:55:05.283228 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:05.283189    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7604-23227a"]
Sep 10 15:55:05.283670 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:05.283558    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb59db5-e927-4c98-a824-256619e14442" containerName="authorino"
Sep 10 15:55:05.283670 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:05.283571    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb59db5-e927-4c98-a824-256619e14442" containerName="authorino"
Sep 10 15:55:05.283670 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:05.283648    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cb59db5-e927-4c98-a824-256619e14442" containerName="authorino"
Sep 10 15:55:05.286615 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:05.286597    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7604-23227a"
Sep 10 15:55:05.288706 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:05.288676    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-j2pv4\""
Sep 10 15:55:05.294956 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:05.294929    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7604-23227a"]
Sep 10 15:55:05.427738 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:05.427698    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xgbb\" (UniqueName: \"kubernetes.io/projected/6908a451-77aa-46f6-ba07-1c44f2b19c23-kube-api-access-4xgbb\") pod \"test-curl-7604-23227a\" (UID: \"6908a451-77aa-46f6-ba07-1c44f2b19c23\") " pod="opendatahub/test-curl-7604-23227a"
Sep 10 15:55:05.528916 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:05.528880    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xgbb\" (UniqueName: \"kubernetes.io/projected/6908a451-77aa-46f6-ba07-1c44f2b19c23-kube-api-access-4xgbb\") pod \"test-curl-7604-23227a\" (UID: \"6908a451-77aa-46f6-ba07-1c44f2b19c23\") " pod="opendatahub/test-curl-7604-23227a"
Sep 10 15:55:05.539402 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:05.539328    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xgbb\" (UniqueName: \"kubernetes.io/projected/6908a451-77aa-46f6-ba07-1c44f2b19c23-kube-api-access-4xgbb\") pod \"test-curl-7604-23227a\" (UID: \"6908a451-77aa-46f6-ba07-1c44f2b19c23\") " pod="opendatahub/test-curl-7604-23227a"
Sep 10 15:55:05.597103 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:05.597063    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7604-23227a"
Sep 10 15:55:05.869496 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:05.869472    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7604-23227a"]
Sep 10 15:55:05.871495 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:55:05.871462    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6908a451_77aa_46f6_ba07_1c44f2b19c23.slice/crio-7626cafbf31c3dbc1ac3bc2e5d3ec57e01dbf835828a181008b530f43c3c8848 WatchSource:0}: Error finding container 7626cafbf31c3dbc1ac3bc2e5d3ec57e01dbf835828a181008b530f43c3c8848: Status 404 returned error can't find the container with id 7626cafbf31c3dbc1ac3bc2e5d3ec57e01dbf835828a181008b530f43c3c8848
Sep 10 15:55:06.752800 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:06.752754    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7604-23227a" event={"ID":"6908a451-77aa-46f6-ba07-1c44f2b19c23","Type":"ContainerStarted","Data":"7626cafbf31c3dbc1ac3bc2e5d3ec57e01dbf835828a181008b530f43c3c8848"}
Sep 10 15:55:08.766022 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:08.765982    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7604-23227a" event={"ID":"6908a451-77aa-46f6-ba07-1c44f2b19c23","Type":"ContainerStarted","Data":"ff692cd96e3637a00d9f059a232ec8cf25ccaccad8e0212d4c2be2dfca91483e"}
Sep 10 15:55:08.792499 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:08.792439    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7604-23227a" podStartSLOduration=1.594784956 podStartE2EDuration="3.792423009s" totalImagesPullingTime="2.197638053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:08.79021789 +0000 UTC m=+781.462627974" watchObservedRunningTime="2026-09-10 15:55:08.792423009 +0000 UTC m=+781.464833091"
Sep 10 15:55:09.354873 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:09.354833    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7604-23227a"]
Sep 10 15:55:09.358953 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:09.358926    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7604-23227a"]
Sep 10 15:55:09.473319 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:09.473223    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7604-134fe1"]
Sep 10 15:55:09.473603 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:09.473590    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6908a451-77aa-46f6-ba07-1c44f2b19c23" containerName="test-curl-7604-23227a"
Sep 10 15:55:09.473649 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:09.473605    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6908a451-77aa-46f6-ba07-1c44f2b19c23" containerName="test-curl-7604-23227a"
Sep 10 15:55:09.473688 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:09.473674    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6908a451-77aa-46f6-ba07-1c44f2b19c23" containerName="test-curl-7604-23227a"
Sep 10 15:55:09.476360 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:09.476344    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7604-134fe1"
Sep 10 15:55:09.482605 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:09.482549    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7604-134fe1"]
Sep 10 15:55:09.566245 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:09.566204    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n7rj\" (UniqueName: \"kubernetes.io/projected/3fe1f2e0-eb0a-4690-894a-026e4655961b-kube-api-access-8n7rj\") pod \"test-curl-7604-134fe1\" (UID: \"3fe1f2e0-eb0a-4690-894a-026e4655961b\") " pod="opendatahub/test-curl-7604-134fe1"
Sep 10 15:55:09.667029 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:09.666994    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n7rj\" (UniqueName: \"kubernetes.io/projected/3fe1f2e0-eb0a-4690-894a-026e4655961b-kube-api-access-8n7rj\") pod \"test-curl-7604-134fe1\" (UID: \"3fe1f2e0-eb0a-4690-894a-026e4655961b\") " pod="opendatahub/test-curl-7604-134fe1"
Sep 10 15:55:09.674319 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:09.674282    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n7rj\" (UniqueName: \"kubernetes.io/projected/3fe1f2e0-eb0a-4690-894a-026e4655961b-kube-api-access-8n7rj\") pod \"test-curl-7604-134fe1\" (UID: \"3fe1f2e0-eb0a-4690-894a-026e4655961b\") " pod="opendatahub/test-curl-7604-134fe1"
Sep 10 15:55:09.771051 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:55:09.770970    2572 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7604-23227a" container="test-curl-7604-23227a"
Sep 10 15:55:09.786874 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:09.786851    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7604-134fe1"
Sep 10 15:55:10.056137 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:10.056076    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7604-134fe1"]
Sep 10 15:55:10.058364 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:55:10.058334    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fe1f2e0_eb0a_4690_894a_026e4655961b.slice/crio-e1f11421dbfd7425224f38a93b9a86103ed4c3e5cf3d3cdf39ec7b460e64c3b0 WatchSource:0}: Error finding container e1f11421dbfd7425224f38a93b9a86103ed4c3e5cf3d3cdf39ec7b460e64c3b0: Status 404 returned error can't find the container with id e1f11421dbfd7425224f38a93b9a86103ed4c3e5cf3d3cdf39ec7b460e64c3b0
Sep 10 15:55:10.777367 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:10.777323    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7604-134fe1" event={"ID":"3fe1f2e0-eb0a-4690-894a-026e4655961b","Type":"ContainerStarted","Data":"fd37066b9b349968ff50d589d9f1af3b70e09884462281b4a93365fb06a67dfd"}
Sep 10 15:55:10.777367 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:10.777364    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7604-23227a" podUID="6908a451-77aa-46f6-ba07-1c44f2b19c23" containerName="test-curl-7604-23227a" containerID="cri-o://ff692cd96e3637a00d9f059a232ec8cf25ccaccad8e0212d4c2be2dfca91483e" gracePeriod=2
Sep 10 15:55:10.777866 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:10.777376    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7604-134fe1" event={"ID":"3fe1f2e0-eb0a-4690-894a-026e4655961b","Type":"ContainerStarted","Data":"e1f11421dbfd7425224f38a93b9a86103ed4c3e5cf3d3cdf39ec7b460e64c3b0"}
Sep 10 15:55:10.791789 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:10.791732    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7604-134fe1" podStartSLOduration=1.791696116 podStartE2EDuration="1.791696116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:10.79082778 +0000 UTC m=+783.463237854" watchObservedRunningTime="2026-09-10 15:55:10.791696116 +0000 UTC m=+783.464106200"
Sep 10 15:55:11.392659 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:11.392611    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7604-134fe1"]
Sep 10 15:55:11.396926 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:11.396891    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7604-134fe1"]
Sep 10 15:55:11.783514 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:55:11.783474    2572 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7604-134fe1" container="test-curl-7604-134fe1"
Sep 10 15:55:12.189849 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.189816    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"]
Sep 10 15:55:12.190278 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.190242    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe1f2e0-eb0a-4690-894a-026e4655961b" containerName="test-curl-7604-134fe1"
Sep 10 15:55:12.190278 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.190277    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe1f2e0-eb0a-4690-894a-026e4655961b" containerName="test-curl-7604-134fe1"
Sep 10 15:55:12.190510 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.190414    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fe1f2e0-eb0a-4690-894a-026e4655961b" containerName="test-curl-7604-134fe1"
Sep 10 15:55:12.193288 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.193269    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.195380 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.195361    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 15:55:12.195503 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.195379    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-314824-openshift-default-dockercfg-jpczw\""
Sep 10 15:55:12.205343 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.205316    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"]
Sep 10 15:55:12.289552 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.289517    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7j9z\" (UniqueName: \"kubernetes.io/projected/da1000a4-2fc8-47f2-9423-689b97933a5a-kube-api-access-k7j9z\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.289757 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.289560    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-credential-socket\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.289757 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.289586    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-workload-socket\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.289757 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.289645    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-envoy\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.289757 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.289697    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1000a4-2fc8-47f2-9423-689b97933a5a-istiod-ca-cert\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.289757 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.289729    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-podinfo\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.290020 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.289764    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-data\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.290020 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.289800    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-workload-certs\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.290020 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.289832    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-token\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.391288 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.391247    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-workload-socket\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.391288 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.391288    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-envoy\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.391551 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.391334    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1000a4-2fc8-47f2-9423-689b97933a5a-istiod-ca-cert\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.391551 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.391361    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-podinfo\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.391551 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.391391    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-data\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.391551 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.391427    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-workload-certs\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.391765 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.391672    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-workload-socket\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.391765 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.391714    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-token\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.391864 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.391820    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7j9z\" (UniqueName: \"kubernetes.io/projected/da1000a4-2fc8-47f2-9423-689b97933a5a-kube-api-access-k7j9z\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.391864 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.391832    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-data\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.391961 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.391869    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-credential-socket\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.392028 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.392007    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-workload-certs\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.392214 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.392193    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-credential-socket\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.392285 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.392224    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1000a4-2fc8-47f2-9423-689b97933a5a-istiod-ca-cert\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.393764 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.393741    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-envoy\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.394201 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.394172    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-podinfo\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.400173 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.400143    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7j9z\" (UniqueName: \"kubernetes.io/projected/da1000a4-2fc8-47f2-9423-689b97933a5a-kube-api-access-k7j9z\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.400771 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.400750    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-token\") pod \"e2e-autores-314824-openshift-default-7fb76f75d6-rfst8\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") " pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.505070 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.504970    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:12.785204 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.784938    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"]
Sep 10 15:55:12.786194 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:55:12.786166    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1000a4_2fc8_47f2_9423_689b97933a5a.slice/crio-3a9b0a3ab4d0fcffada16c651ba900a882ae2781d8c6760864ac3a5cceaa4aed WatchSource:0}: Error finding container 3a9b0a3ab4d0fcffada16c651ba900a882ae2781d8c6760864ac3a5cceaa4aed: Status 404 returned error can't find the container with id 3a9b0a3ab4d0fcffada16c651ba900a882ae2781d8c6760864ac3a5cceaa4aed
Sep 10 15:55:12.788897 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.788869    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7604-134fe1" podUID="3fe1f2e0-eb0a-4690-894a-026e4655961b" containerName="test-curl-7604-134fe1" containerID="cri-o://fd37066b9b349968ff50d589d9f1af3b70e09884462281b4a93365fb06a67dfd" gracePeriod=2
Sep 10 15:55:12.931063 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:12.931034    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7604-23227a"
Sep 10 15:55:13.098977 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:13.098938    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xgbb\" (UniqueName: \"kubernetes.io/projected/6908a451-77aa-46f6-ba07-1c44f2b19c23-kube-api-access-4xgbb\") pod \"6908a451-77aa-46f6-ba07-1c44f2b19c23\" (UID: \"6908a451-77aa-46f6-ba07-1c44f2b19c23\") "
Sep 10 15:55:13.101161 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:13.101128    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6908a451-77aa-46f6-ba07-1c44f2b19c23-kube-api-access-4xgbb" (OuterVolumeSpecName: "kube-api-access-4xgbb") pod "6908a451-77aa-46f6-ba07-1c44f2b19c23" (UID: "6908a451-77aa-46f6-ba07-1c44f2b19c23"). InnerVolumeSpecName "kube-api-access-4xgbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:55:13.200271 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:13.200239    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xgbb\" (UniqueName: \"kubernetes.io/projected/6908a451-77aa-46f6-ba07-1c44f2b19c23-kube-api-access-4xgbb\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:55:13.797313 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:13.797208    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8" event={"ID":"da1000a4-2fc8-47f2-9423-689b97933a5a","Type":"ContainerStarted","Data":"3a9b0a3ab4d0fcffada16c651ba900a882ae2781d8c6760864ac3a5cceaa4aed"}
Sep 10 15:55:13.799033 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:13.799002    2572 generic.go:356] "Generic (PLEG): container finished" podID="6908a451-77aa-46f6-ba07-1c44f2b19c23" containerID="ff692cd96e3637a00d9f059a232ec8cf25ccaccad8e0212d4c2be2dfca91483e" exitCode=137
Sep 10 15:55:13.799185 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:13.799085    2572 scope.go:117] "RemoveContainer" containerID="ff692cd96e3637a00d9f059a232ec8cf25ccaccad8e0212d4c2be2dfca91483e"
Sep 10 15:55:13.799185 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:13.799087    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7604-23227a"
Sep 10 15:55:13.812922 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:13.812902    2572 scope.go:117] "RemoveContainer" containerID="ff692cd96e3637a00d9f059a232ec8cf25ccaccad8e0212d4c2be2dfca91483e"
Sep 10 15:55:13.813142 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:55:13.813081    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7604-23227a\" is forbidden: User \"system:node:ip-10-0-134-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-36.ec2.internal' and this object" podUID="6908a451-77aa-46f6-ba07-1c44f2b19c23" pod="opendatahub/test-curl-7604-23227a"
Sep 10 15:55:13.813293 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:55:13.813272    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff692cd96e3637a00d9f059a232ec8cf25ccaccad8e0212d4c2be2dfca91483e\": container with ID starting with ff692cd96e3637a00d9f059a232ec8cf25ccaccad8e0212d4c2be2dfca91483e not found: ID does not exist" containerID="ff692cd96e3637a00d9f059a232ec8cf25ccaccad8e0212d4c2be2dfca91483e"
Sep 10 15:55:13.813389 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:13.813321    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff692cd96e3637a00d9f059a232ec8cf25ccaccad8e0212d4c2be2dfca91483e"} err="failed to get container status \"ff692cd96e3637a00d9f059a232ec8cf25ccaccad8e0212d4c2be2dfca91483e\": rpc error: code = NotFound desc = could not find container \"ff692cd96e3637a00d9f059a232ec8cf25ccaccad8e0212d4c2be2dfca91483e\": container with ID starting with ff692cd96e3637a00d9f059a232ec8cf25ccaccad8e0212d4c2be2dfca91483e not found: ID does not exist"
Sep 10 15:55:13.995349 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:13.995312    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6908a451-77aa-46f6-ba07-1c44f2b19c23" path="/var/lib/kubelet/pods/6908a451-77aa-46f6-ba07-1c44f2b19c23/volumes"
Sep 10 15:55:15.523960 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:15.523918    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:15.524271 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:15.523999    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:15.524271 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:15.524027    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:15.610202 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:15.610168    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7604-134fe1"
Sep 10 15:55:15.724108 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:15.724015    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n7rj\" (UniqueName: \"kubernetes.io/projected/3fe1f2e0-eb0a-4690-894a-026e4655961b-kube-api-access-8n7rj\") pod \"3fe1f2e0-eb0a-4690-894a-026e4655961b\" (UID: \"3fe1f2e0-eb0a-4690-894a-026e4655961b\") "
Sep 10 15:55:15.726140 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:15.726106    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe1f2e0-eb0a-4690-894a-026e4655961b-kube-api-access-8n7rj" (OuterVolumeSpecName: "kube-api-access-8n7rj") pod "3fe1f2e0-eb0a-4690-894a-026e4655961b" (UID: "3fe1f2e0-eb0a-4690-894a-026e4655961b"). InnerVolumeSpecName "kube-api-access-8n7rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:55:15.814449 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:15.814414    2572 generic.go:356] "Generic (PLEG): container finished" podID="3fe1f2e0-eb0a-4690-894a-026e4655961b" containerID="fd37066b9b349968ff50d589d9f1af3b70e09884462281b4a93365fb06a67dfd" exitCode=137
Sep 10 15:55:15.814622 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:15.814453    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7604-134fe1"
Sep 10 15:55:15.814622 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:15.814512    2572 scope.go:117] "RemoveContainer" containerID="fd37066b9b349968ff50d589d9f1af3b70e09884462281b4a93365fb06a67dfd"
Sep 10 15:55:15.816554 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:15.816527    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8" event={"ID":"da1000a4-2fc8-47f2-9423-689b97933a5a","Type":"ContainerStarted","Data":"de626fa1f5323bdf578bd32a3757c0aabdfb58a73ba62df235654db6db609281"}
Sep 10 15:55:15.825156 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:15.825138    2572 scope.go:117] "RemoveContainer" containerID="fd37066b9b349968ff50d589d9f1af3b70e09884462281b4a93365fb06a67dfd"
Sep 10 15:55:15.825514 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:55:15.825487    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd37066b9b349968ff50d589d9f1af3b70e09884462281b4a93365fb06a67dfd\": container with ID starting with fd37066b9b349968ff50d589d9f1af3b70e09884462281b4a93365fb06a67dfd not found: ID does not exist" containerID="fd37066b9b349968ff50d589d9f1af3b70e09884462281b4a93365fb06a67dfd"
Sep 10 15:55:15.825630 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:15.825521    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd37066b9b349968ff50d589d9f1af3b70e09884462281b4a93365fb06a67dfd"} err="failed to get container status \"fd37066b9b349968ff50d589d9f1af3b70e09884462281b4a93365fb06a67dfd\": rpc error: code = NotFound desc = could not find container \"fd37066b9b349968ff50d589d9f1af3b70e09884462281b4a93365fb06a67dfd\": container with ID starting with fd37066b9b349968ff50d589d9f1af3b70e09884462281b4a93365fb06a67dfd not found: ID does not exist"
Sep 10 15:55:15.825630 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:15.825591    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n7rj\" (UniqueName: \"kubernetes.io/projected/3fe1f2e0-eb0a-4690-894a-026e4655961b-kube-api-access-8n7rj\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:55:15.841870 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:55:15.841837    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7604-134fe1\" is forbidden: User \"system:node:ip-10-0-134-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-36.ec2.internal' and this object" podUID="3fe1f2e0-eb0a-4690-894a-026e4655961b" pod="opendatahub/test-curl-7604-134fe1"
Sep 10 15:55:15.842215 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:15.842177    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8" podStartSLOduration=1.130935968 podStartE2EDuration="3.842164994s" totalImagesPullingTime="2.711229026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:15.839712439 +0000 UTC m=+788.512122522" watchObservedRunningTime="2026-09-10 15:55:15.842164994 +0000 UTC m=+788.514575133"
Sep 10 15:55:15.998382 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:15.993774    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fe1f2e0-eb0a-4690-894a-026e4655961b" path="/var/lib/kubelet/pods/3fe1f2e0-eb0a-4690-894a-026e4655961b/volumes"
Sep 10 15:55:16.505943 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:16.505906    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:16.511060 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:16.511033    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:16.825490 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:16.825453    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:16.827020 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:16.826993    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:32.154790 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.154730    2572 patch_prober.go:28] interesting pod/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": read tcp 10.134.0.2:59772->10.134.0.42:15021: read: connection reset by peer" start-of-body=
Sep 10 15:55:32.155201 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.154832    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8" podUID="da1000a4-2fc8-47f2-9423-689b97933a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": read tcp 10.134.0.2:59772->10.134.0.42:15021: read: connection reset by peer"
Sep 10 15:55:32.299086 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.299049    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"]
Sep 10 15:55:32.301451 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.301433    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.305400 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.305372    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5d6f34-openshift-default-dockercfg-zv4x6\""
Sep 10 15:55:32.317407 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.317373    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"]
Sep 10 15:55:32.388007 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.387915    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-workload-certs\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.388007 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.387972    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-token\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.388222 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.388003    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf25b\" (UniqueName: \"kubernetes.io/projected/11f92f44-7e15-4416-8383-ba28e233d3b3-kube-api-access-lf25b\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.388222 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.388051    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-credential-socket\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.388222 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.388093    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-workload-socket\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.388222 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.388125    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f92f44-7e15-4416-8383-ba28e233d3b3-istiod-ca-cert\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.388405 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.388213    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-podinfo\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.388405 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.388241    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-data\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.388405 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.388275    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-envoy\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.489691 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.489544    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-token\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.489691 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.489585    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf25b\" (UniqueName: \"kubernetes.io/projected/11f92f44-7e15-4416-8383-ba28e233d3b3-kube-api-access-lf25b\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.489691 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.489607    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-credential-socket\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.489691 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.489632    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-workload-socket\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.489691 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.489673    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f92f44-7e15-4416-8383-ba28e233d3b3-istiod-ca-cert\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.490089 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.489712    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-podinfo\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.490089 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.490021    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-data\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.490213 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.490117    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-credential-socket\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.490213 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.490154    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-workload-socket\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.490364 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.490234    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-data\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.490417 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.490085    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-envoy\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.490469 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.490449    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-workload-certs\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.490599 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.490568    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f92f44-7e15-4416-8383-ba28e233d3b3-istiod-ca-cert\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.490745 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.490720    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-workload-certs\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.492942 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.492911    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-envoy\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.493223 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.493201    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-podinfo\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.507758 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.507732    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf25b\" (UniqueName: \"kubernetes.io/projected/11f92f44-7e15-4416-8383-ba28e233d3b3-kube-api-access-lf25b\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.508810 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.508785    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-token\") pod \"e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") " pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.611908 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.611865    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:32.889841 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.889811    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"]
Sep 10 15:55:32.891446 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:55:32.891416    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f92f44_7e15_4416_8383_ba28e233d3b3.slice/crio-cf57b73756bf4418aefd4ea8d409fecf47023b182826b7f3a87ec6a7e4ff759f WatchSource:0}: Error finding container cf57b73756bf4418aefd4ea8d409fecf47023b182826b7f3a87ec6a7e4ff759f: Status 404 returned error can't find the container with id cf57b73756bf4418aefd4ea8d409fecf47023b182826b7f3a87ec6a7e4ff759f
Sep 10 15:55:32.894832 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.894797    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:32.894934 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.894870    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:32.894934 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.894899    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:32.918556 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.918524    2572 generic.go:356] "Generic (PLEG): container finished" podID="da1000a4-2fc8-47f2-9423-689b97933a5a" containerID="de626fa1f5323bdf578bd32a3757c0aabdfb58a73ba62df235654db6db609281" exitCode=0
Sep 10 15:55:32.918699 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.918602    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8" event={"ID":"da1000a4-2fc8-47f2-9423-689b97933a5a","Type":"ContainerDied","Data":"de626fa1f5323bdf578bd32a3757c0aabdfb58a73ba62df235654db6db609281"}
Sep 10 15:55:32.919295 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.919091    2572 scope.go:117] "RemoveContainer" containerID="de626fa1f5323bdf578bd32a3757c0aabdfb58a73ba62df235654db6db609281"
Sep 10 15:55:32.920177 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.920151    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq" event={"ID":"11f92f44-7e15-4416-8383-ba28e233d3b3","Type":"ContainerStarted","Data":"cf57b73756bf4418aefd4ea8d409fecf47023b182826b7f3a87ec6a7e4ff759f"}
Sep 10 15:55:32.921452 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.921426    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:32.921551 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.921511    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:32.921620 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:32.921561    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:33.931191 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:33.931151    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8" event={"ID":"da1000a4-2fc8-47f2-9423-689b97933a5a","Type":"ContainerStarted","Data":"2156655f191b50fc4ffe170bce1ca4e5db431b31e579bbcbffa165942765541d"}
Sep 10 15:55:33.935095 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:33.935062    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq" event={"ID":"11f92f44-7e15-4416-8383-ba28e233d3b3","Type":"ContainerStarted","Data":"0b93d939af3952b212bce7e917f3727b3a2df12e1579743a3cb5ffe1398760d6"}
Sep 10 15:55:33.974520 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:33.974462    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq" podStartSLOduration=1.974444353 podStartE2EDuration="1.974444353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:33.972468195 +0000 UTC m=+806.644878291" watchObservedRunningTime="2026-09-10 15:55:33.974444353 +0000 UTC m=+806.646854498"
Sep 10 15:55:34.505709 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:34.505669    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:34.505905 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:34.505886    2572 patch_prober.go:28] interesting pod/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 10 15:55:34.505975 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:34.505954    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8" podUID="da1000a4-2fc8-47f2-9423-689b97933a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 10 15:55:34.613057 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:34.613015    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:34.622048 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:34.622001    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:34.943936 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:34.943864    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:34.945735 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:34.945705    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:35.505566 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:35.505522    2572 patch_prober.go:28] interesting pod/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 10 15:55:35.505766 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:35.505601    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8" podUID="da1000a4-2fc8-47f2-9423-689b97933a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 10 15:55:36.506085 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:36.506046    2572 patch_prober.go:28] interesting pod/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 10 15:55:36.506627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:36.506122    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8" podUID="da1000a4-2fc8-47f2-9423-689b97933a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 10 15:55:37.505869 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:37.505834    2572 patch_prober.go:28] interesting pod/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 10 15:55:37.506069 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:37.505893    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8" podUID="da1000a4-2fc8-47f2-9423-689b97933a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 10 15:55:38.505538 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:38.505502    2572 patch_prober.go:28] interesting pod/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 10 15:55:38.505957 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:38.505573    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8" podUID="da1000a4-2fc8-47f2-9423-689b97933a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 10 15:55:39.509727 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:39.509695    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:39.510110 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:39.509944    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:39.510755 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:39.510737    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:55:39.729129 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:39.729090    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-941c6070-jdqs7"]
Sep 10 15:55:39.732616 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:39.732597    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-941c6070-jdqs7"
Sep 10 15:55:39.734863 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:39.734837    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\""
Sep 10 15:55:39.735161 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:39.735140    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 15:55:39.741518 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:39.741491    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-941c6070-jdqs7"]
Sep 10 15:55:39.864941 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:39.864902    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f154365d-07e1-428e-849b-60baaebfdf3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-941c6070-jdqs7\" (UID: \"f154365d-07e1-428e-849b-60baaebfdf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-941c6070-jdqs7"
Sep 10 15:55:39.966084 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:39.966043    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f154365d-07e1-428e-849b-60baaebfdf3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-941c6070-jdqs7\" (UID: \"f154365d-07e1-428e-849b-60baaebfdf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-941c6070-jdqs7"
Sep 10 15:55:39.966877 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:39.966851    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f154365d-07e1-428e-849b-60baaebfdf3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-941c6070-jdqs7\" (UID: \"f154365d-07e1-428e-849b-60baaebfdf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-941c6070-jdqs7"
Sep 10 15:55:40.044273 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:40.044227    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-941c6070-jdqs7"
Sep 10 15:55:40.315883 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:40.315856    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-941c6070-jdqs7"]
Sep 10 15:55:40.318003 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:55:40.317972    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf154365d_07e1_428e_849b_60baaebfdf3a.slice/crio-bf4357d02b5489db307d942b75e8cc991674a66c28f2d290c17b9771abf15a30 WatchSource:0}: Error finding container bf4357d02b5489db307d942b75e8cc991674a66c28f2d290c17b9771abf15a30: Status 404 returned error can't find the container with id bf4357d02b5489db307d942b75e8cc991674a66c28f2d290c17b9771abf15a30
Sep 10 15:55:40.978464 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:40.978426    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-941c6070-jdqs7" event={"ID":"f154365d-07e1-428e-849b-60baaebfdf3a","Type":"ContainerStarted","Data":"bf4357d02b5489db307d942b75e8cc991674a66c28f2d290c17b9771abf15a30"}
Sep 10 15:55:43.997350 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:43.997283    2572 generic.go:356] "Generic (PLEG): container finished" podID="f154365d-07e1-428e-849b-60baaebfdf3a" containerID="e011f3ed85c4a5b5e9bbbc8533b47858090643990fa47543dbb6ef66d64cffc4" exitCode=0
Sep 10 15:55:43.997769 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:43.997365    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-941c6070-jdqs7" event={"ID":"f154365d-07e1-428e-849b-60baaebfdf3a","Type":"ContainerDied","Data":"e011f3ed85c4a5b5e9bbbc8533b47858090643990fa47543dbb6ef66d64cffc4"}
Sep 10 15:55:45.173160 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:45.173116    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-941c6070-jdqs7"
Sep 10 15:55:45.337559 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:45.337512    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f154365d-07e1-428e-849b-60baaebfdf3a-maas-api-service-ca\") pod \"f154365d-07e1-428e-849b-60baaebfdf3a\" (UID: \"f154365d-07e1-428e-849b-60baaebfdf3a\") "
Sep 10 15:55:45.338177 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:45.338144    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f154365d-07e1-428e-849b-60baaebfdf3a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f154365d-07e1-428e-849b-60baaebfdf3a" (UID: "f154365d-07e1-428e-849b-60baaebfdf3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:55:45.438681 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:45.438630    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f154365d-07e1-428e-849b-60baaebfdf3a-maas-api-service-ca\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:55:46.009801 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:46.009773    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-941c6070-jdqs7"
Sep 10 15:55:46.009801 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:46.009785    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-941c6070-jdqs7" event={"ID":"f154365d-07e1-428e-849b-60baaebfdf3a","Type":"ContainerDied","Data":"bf4357d02b5489db307d942b75e8cc991674a66c28f2d290c17b9771abf15a30"}
Sep 10 15:55:46.009996 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:46.009830    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf4357d02b5489db307d942b75e8cc991674a66c28f2d290c17b9771abf15a30"
Sep 10 15:55:50.441327 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.441269    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"]
Sep 10 15:55:50.441748 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.441631    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f154365d-07e1-428e-849b-60baaebfdf3a" containerName="revoke-keys"
Sep 10 15:55:50.441748 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.441643    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f154365d-07e1-428e-849b-60baaebfdf3a" containerName="revoke-keys"
Sep 10 15:55:50.441748 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.441714    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f154365d-07e1-428e-849b-60baaebfdf3a" containerName="revoke-keys"
Sep 10 15:55:50.445009 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.444987    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.447294 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.447272    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-0ce14d90-kserve-self-signed-certs\""
Sep 10 15:55:50.454858 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.454826    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"]
Sep 10 15:55:50.591513 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.591468    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-model-cache\") pod \"e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") " pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.591717 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.591527    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-home\") pod \"e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") " pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.591717 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.591557    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-tmp-dir\") pod \"e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") " pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.591717 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.591602    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77548a93-6c9a-4e79-8751-953f66737362-tls-certs\") pod \"e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") " pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.591717 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.591655    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-dshm\") pod \"e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") " pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.591884 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.591719    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv2pc\" (UniqueName: \"kubernetes.io/projected/77548a93-6c9a-4e79-8751-953f66737362-kube-api-access-gv2pc\") pod \"e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") " pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.693330 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.693195    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-model-cache\") pod \"e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") " pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.693330 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.693270    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-home\") pod \"e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") " pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.693330 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.693328    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-tmp-dir\") pod \"e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") " pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.693617 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.693346    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77548a93-6c9a-4e79-8751-953f66737362-tls-certs\") pod \"e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") " pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.693617 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.693375    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-dshm\") pod \"e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") " pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.693617 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.693413    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv2pc\" (UniqueName: \"kubernetes.io/projected/77548a93-6c9a-4e79-8751-953f66737362-kube-api-access-gv2pc\") pod \"e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") " pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.693759 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.693717    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-model-cache\") pod \"e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") " pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.693816 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.693784    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-tmp-dir\") pod \"e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") " pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.693935 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.693916    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-home\") pod \"e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") " pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.695811 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.695783    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-dshm\") pod \"e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") " pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.696029 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.696005    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77548a93-6c9a-4e79-8751-953f66737362-tls-certs\") pod \"e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") " pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.702349 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.702286    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv2pc\" (UniqueName: \"kubernetes.io/projected/77548a93-6c9a-4e79-8751-953f66737362-kube-api-access-gv2pc\") pod \"e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") " pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.756142 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.756101    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:55:50.986534 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.986434    2572 patch_prober.go:28] interesting pod/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 15:55:50.986717 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:50.986533    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq" podUID="11f92f44-7e15-4416-8383-ba28e233d3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 15:55:51.032207 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:51.032175    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"]
Sep 10 15:55:51.033847 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:55:51.033819    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77548a93_6c9a_4e79_8751_953f66737362.slice/crio-240afee1b95fb3e14ab5b4dede9b381bb8db49e19fec8f33b21553ee106ba06c WatchSource:0}: Error finding container 240afee1b95fb3e14ab5b4dede9b381bb8db49e19fec8f33b21553ee106ba06c: Status 404 returned error can't find the container with id 240afee1b95fb3e14ab5b4dede9b381bb8db49e19fec8f33b21553ee106ba06c
Sep 10 15:55:51.051379 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:51.051339    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx" event={"ID":"77548a93-6c9a-4e79-8751-953f66737362","Type":"ContainerStarted","Data":"240afee1b95fb3e14ab5b4dede9b381bb8db49e19fec8f33b21553ee106ba06c"}
Sep 10 15:55:52.059490 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:52.059446    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx" event={"ID":"77548a93-6c9a-4e79-8751-953f66737362","Type":"ContainerStarted","Data":"553bc30f2f9044181a2b1d4d3e173cf46360653f5a0ea6f2322b6c7092c6273a"}
Sep 10 15:55:52.081777 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:52.081711    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx" podStartSLOduration=2.081693528 podStartE2EDuration="2.081693528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:52.078671641 +0000 UTC m=+824.751081726" watchObservedRunningTime="2026-09-10 15:55:52.081693528 +0000 UTC m=+824.754103631"
Sep 10 15:55:53.066993 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:53.066960    2572 generic.go:356] "Generic (PLEG): container finished" podID="11f92f44-7e15-4416-8383-ba28e233d3b3" containerID="0b93d939af3952b212bce7e917f3727b3a2df12e1579743a3cb5ffe1398760d6" exitCode=0
Sep 10 15:55:53.067544 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:53.067045    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq" event={"ID":"11f92f44-7e15-4416-8383-ba28e233d3b3","Type":"ContainerDied","Data":"0b93d939af3952b212bce7e917f3727b3a2df12e1579743a3cb5ffe1398760d6"}
Sep 10 15:55:53.067617 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:53.067603    2572 scope.go:117] "RemoveContainer" containerID="0b93d939af3952b212bce7e917f3727b3a2df12e1579743a3cb5ffe1398760d6"
Sep 10 15:55:53.068612 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:53.068586    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:53.068674 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:53.068653    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:53.068714 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:53.068685    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:54.074489 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:54.074449    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq" event={"ID":"11f92f44-7e15-4416-8383-ba28e233d3b3","Type":"ContainerStarted","Data":"406aef7c0a20146ef807d65a526cb50ae82626bca802974a0054e89c97d7cedf"}
Sep 10 15:55:54.612952 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:54.612861    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:55:54.613114 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:54.613063    2572 patch_prober.go:28] interesting pod/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 15:55:54.613157 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:54.613117    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq" podUID="11f92f44-7e15-4416-8383-ba28e233d3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 15:55:55.613138 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:55.613096    2572 patch_prober.go:28] interesting pod/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 15:55:55.613667 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:55.613180    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq" podUID="11f92f44-7e15-4416-8383-ba28e233d3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 15:55:56.612788 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:56.612750    2572 patch_prober.go:28] interesting pod/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 15:55:56.612966 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:56.612828    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq" podUID="11f92f44-7e15-4416-8383-ba28e233d3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 15:55:56.727357 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:56.727317    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-941c6070-jdqs7"]
Sep 10 15:55:56.731082 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:56.731046    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-941c6070-jdqs7"]
Sep 10 15:55:57.612313 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:57.612271    2572 patch_prober.go:28] interesting pod/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 15:55:57.612500 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:57.612357    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq" podUID="11f92f44-7e15-4416-8383-ba28e233d3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 15:55:57.942852 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:57.942744    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"]
Sep 10 15:55:57.945928 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:57.945901    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:57.950591 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:57.950570    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-0fcd7b0f-gw-openshift-default-dockercfg-tkrhl\""
Sep 10 15:55:57.965141 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:57.965112    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-credential-socket\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:57.965277 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:57.965156    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-envoy\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:57.965277 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:57.965173    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-token\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:57.965277 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:57.965218    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61dfee7-c61d-4bde-a32c-746abe930ac7-istiod-ca-cert\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:57.965277 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:57.965257    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shg7r\" (UniqueName: \"kubernetes.io/projected/e61dfee7-c61d-4bde-a32c-746abe930ac7-kube-api-access-shg7r\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:57.965451 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:57.965287    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-podinfo\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:57.965451 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:57.965334    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-data\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:57.965451 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:57.965370    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-workload-socket\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:57.965451 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:57.965441    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-workload-certs\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:57.975277 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:57.975238    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"]
Sep 10 15:55:57.993673 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:57.993639    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f154365d-07e1-428e-849b-60baaebfdf3a" path="/var/lib/kubelet/pods/f154365d-07e1-428e-849b-60baaebfdf3a/volumes"
Sep 10 15:55:58.065793 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.065755    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-data\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:58.065971 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.065803    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-workload-socket\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:58.066034 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.065979    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-workload-certs\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:58.066091 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.066065    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-credential-socket\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:58.066151 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.066100    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-data\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:58.066151 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.066124    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-envoy\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:58.066262 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.066231    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-token\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:58.066262 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.066241    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-workload-socket\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:58.066412 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.066263    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61dfee7-c61d-4bde-a32c-746abe930ac7-istiod-ca-cert\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:58.066412 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.066340    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shg7r\" (UniqueName: \"kubernetes.io/projected/e61dfee7-c61d-4bde-a32c-746abe930ac7-kube-api-access-shg7r\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:58.066412 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.066379    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-podinfo\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:58.066580 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.066478    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-workload-certs\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:58.066720 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.066698    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-credential-socket\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:58.067006 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.066985    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61dfee7-c61d-4bde-a32c-746abe930ac7-istiod-ca-cert\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:58.068350 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.068331    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-envoy\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:58.068627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.068610    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-podinfo\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:58.075210 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.075189    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-token\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:58.077160 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.077137    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shg7r\" (UniqueName: \"kubernetes.io/projected/e61dfee7-c61d-4bde-a32c-746abe930ac7-kube-api-access-shg7r\") pod \"e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") " pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:58.256876 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.256776    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:58.545349 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.545202    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"]
Sep 10 15:55:58.549593 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:55:58.549558    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61dfee7_c61d_4bde_a32c_746abe930ac7.slice/crio-a5ec6160906456377a5aabaa8019df8b1c7808d7160edb568352d34ad5523c9a WatchSource:0}: Error finding container a5ec6160906456377a5aabaa8019df8b1c7808d7160edb568352d34ad5523c9a: Status 404 returned error can't find the container with id a5ec6160906456377a5aabaa8019df8b1c7808d7160edb568352d34ad5523c9a
Sep 10 15:55:58.551947 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.551909    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:58.552053 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.551983    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:58.552053 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.552015    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:58.612882 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.612853    2572 patch_prober.go:28] interesting pod/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 15:55:58.613002 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:58.612904    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq" podUID="11f92f44-7e15-4416-8383-ba28e233d3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 15:55:59.104352 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:59.104290    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc" event={"ID":"e61dfee7-c61d-4bde-a32c-746abe930ac7","Type":"ContainerStarted","Data":"047cc06f895830d8d64c08686838a7a4a9f283b8cb220f57e5bca8e86f6bc5f9"}
Sep 10 15:55:59.104754 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:59.104364    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc" event={"ID":"e61dfee7-c61d-4bde-a32c-746abe930ac7","Type":"ContainerStarted","Data":"a5ec6160906456377a5aabaa8019df8b1c7808d7160edb568352d34ad5523c9a"}
Sep 10 15:55:59.135092 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:59.135018    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc" podStartSLOduration=2.134986709 podStartE2EDuration="2.134986709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:59.131148119 +0000 UTC m=+831.803558227" watchObservedRunningTime="2026-09-10 15:55:59.134986709 +0000 UTC m=+831.807396793"
Sep 10 15:55:59.257044 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:59.257000    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:59.276007 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:59.275976    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:55:59.612273 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:59.612232    2572 patch_prober.go:28] interesting pod/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 15:55:59.612447 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:55:59.612289    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq" podUID="11f92f44-7e15-4416-8383-ba28e233d3b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 15:56:00.111221 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:00.111186    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:56:00.112329 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:00.112283    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:56:00.615928 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:00.615901    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:56:00.616335 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:00.616311    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:56:00.617070 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:00.617052    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 15:56:00.756611 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:00.756578    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:56:00.756781 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:00.756637    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:56:00.769451 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:00.769425    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:56:01.128081 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:01.128050    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:56:08.238725 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.238669    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"]
Sep 10 15:56:08.243956 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.243925    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.245962 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.245942    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-314824\"/\"autores-model-314824-kserve-self-signed-certs\""
Sep 10 15:56:08.246909 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.246851    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-314824\"/\"kube-root-ca.crt\""
Sep 10 15:56:08.246909 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.246897    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-314824\"/\"openshift-service-ca.crt\""
Sep 10 15:56:08.247079 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.246911    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-314824\"/\"default-dockercfg-w7n6w\""
Sep 10 15:56:08.253276 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.253251    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"]
Sep 10 15:56:08.374207 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.374174    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-dshm\") pod \"autores-model-314824-kserve-779bccc899-hzwbj\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") " pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.374207 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.374206    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-tmp-dir\") pod \"autores-model-314824-kserve-779bccc899-hzwbj\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") " pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.374461 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.374229    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-model-cache\") pod \"autores-model-314824-kserve-779bccc899-hzwbj\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") " pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.374461 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.374252    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81add4b2-a2e8-4729-8c3d-96d3b1421539-tls-certs\") pod \"autores-model-314824-kserve-779bccc899-hzwbj\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") " pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.374461 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.374437    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4mh8\" (UniqueName: \"kubernetes.io/projected/81add4b2-a2e8-4729-8c3d-96d3b1421539-kube-api-access-l4mh8\") pod \"autores-model-314824-kserve-779bccc899-hzwbj\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") " pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.374578 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.374468    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-home\") pod \"autores-model-314824-kserve-779bccc899-hzwbj\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") " pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.475218 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.475175    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-dshm\") pod \"autores-model-314824-kserve-779bccc899-hzwbj\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") " pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.475218 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.475219    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-tmp-dir\") pod \"autores-model-314824-kserve-779bccc899-hzwbj\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") " pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.475511 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.475247    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-model-cache\") pod \"autores-model-314824-kserve-779bccc899-hzwbj\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") " pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.475511 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.475281    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81add4b2-a2e8-4729-8c3d-96d3b1421539-tls-certs\") pod \"autores-model-314824-kserve-779bccc899-hzwbj\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") " pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.475511 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.475406    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4mh8\" (UniqueName: \"kubernetes.io/projected/81add4b2-a2e8-4729-8c3d-96d3b1421539-kube-api-access-l4mh8\") pod \"autores-model-314824-kserve-779bccc899-hzwbj\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") " pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.475511 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.475434    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-home\") pod \"autores-model-314824-kserve-779bccc899-hzwbj\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") " pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.475739 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.475710    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-tmp-dir\") pod \"autores-model-314824-kserve-779bccc899-hzwbj\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") " pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.475803 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.475721    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-model-cache\") pod \"autores-model-314824-kserve-779bccc899-hzwbj\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") " pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.475855 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.475800    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-home\") pod \"autores-model-314824-kserve-779bccc899-hzwbj\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") " pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.477615 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.477594    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-dshm\") pod \"autores-model-314824-kserve-779bccc899-hzwbj\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") " pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.477840 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.477820    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81add4b2-a2e8-4729-8c3d-96d3b1421539-tls-certs\") pod \"autores-model-314824-kserve-779bccc899-hzwbj\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") " pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.485059 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.485031    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4mh8\" (UniqueName: \"kubernetes.io/projected/81add4b2-a2e8-4729-8c3d-96d3b1421539-kube-api-access-l4mh8\") pod \"autores-model-314824-kserve-779bccc899-hzwbj\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") " pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.553834 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.553794    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:08.830828 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:08.830799    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"]
Sep 10 15:56:08.832338 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:56:08.832295    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81add4b2_a2e8_4729_8c3d_96d3b1421539.slice/crio-c66d8474b5e2b277bf4b45dc01cc62696878575b2b5a92468e2d3ba6bcf3c440 WatchSource:0}: Error finding container c66d8474b5e2b277bf4b45dc01cc62696878575b2b5a92468e2d3ba6bcf3c440: Status 404 returned error can't find the container with id c66d8474b5e2b277bf4b45dc01cc62696878575b2b5a92468e2d3ba6bcf3c440
Sep 10 15:56:09.164743 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:09.164650    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj" event={"ID":"81add4b2-a2e8-4729-8c3d-96d3b1421539","Type":"ContainerStarted","Data":"9b8d292592410b7568890496531dc77043c8087d9e7b997e0b207e48122829d6"}
Sep 10 15:56:09.164743 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:09.164697    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj" event={"ID":"81add4b2-a2e8-4729-8c3d-96d3b1421539","Type":"ContainerStarted","Data":"c66d8474b5e2b277bf4b45dc01cc62696878575b2b5a92468e2d3ba6bcf3c440"}
Sep 10 15:56:09.182152 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:09.182096    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj" podStartSLOduration=1.182079503 podStartE2EDuration="1.182079503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:09.180650863 +0000 UTC m=+841.853060951" watchObservedRunningTime="2026-09-10 15:56:09.182079503 +0000 UTC m=+841.854489622"
Sep 10 15:56:18.554435 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:18.554388    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:18.554888 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:18.554453    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:18.567761 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:18.567727    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:19.254339 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:19.254283    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:56:23.256801 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:23.256711    2572 generic.go:356] "Generic (PLEG): container finished" podID="e61dfee7-c61d-4bde-a32c-746abe930ac7" containerID="047cc06f895830d8d64c08686838a7a4a9f283b8cb220f57e5bca8e86f6bc5f9" exitCode=0
Sep 10 15:56:23.256801 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:23.256781    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc" event={"ID":"e61dfee7-c61d-4bde-a32c-746abe930ac7","Type":"ContainerDied","Data":"047cc06f895830d8d64c08686838a7a4a9f283b8cb220f57e5bca8e86f6bc5f9"}
Sep 10 15:56:23.257338 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:23.257326    2572 scope.go:117] "RemoveContainer" containerID="047cc06f895830d8d64c08686838a7a4a9f283b8cb220f57e5bca8e86f6bc5f9"
Sep 10 15:56:23.258509 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:23.258479    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:56:23.258603 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:23.258587    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:56:23.258646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:23.258631    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:56:24.264771 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:24.264735    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc" event={"ID":"e61dfee7-c61d-4bde-a32c-746abe930ac7","Type":"ContainerStarted","Data":"3ef505f1d2b71f3b3a4c1efb6cffe1db66cf3146ea04155b9d6f59873a1ab2e1"}
Sep 10 15:56:25.257646 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:25.257597    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:56:25.257915 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:25.257891    2572 patch_prober.go:28] interesting pod/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 10 15:56:25.257966 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:25.257940    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc" podUID="e61dfee7-c61d-4bde-a32c-746abe930ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 10 15:56:26.257734 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:26.257689    2572 patch_prober.go:28] interesting pod/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 10 15:56:26.258222 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:26.257771    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc" podUID="e61dfee7-c61d-4bde-a32c-746abe930ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 10 15:56:27.258117 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:27.258077    2572 patch_prober.go:28] interesting pod/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 10 15:56:27.258564 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:27.258155    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc" podUID="e61dfee7-c61d-4bde-a32c-746abe930ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 10 15:56:28.257483 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:28.257447    2572 patch_prober.go:28] interesting pod/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 10 15:56:28.257699 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:28.257511    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc" podUID="e61dfee7-c61d-4bde-a32c-746abe930ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 10 15:56:29.257515 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:29.257474    2572 patch_prober.go:28] interesting pod/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 10 15:56:29.257921 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:29.257536    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc" podUID="e61dfee7-c61d-4bde-a32c-746abe930ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 10 15:56:30.111165 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:30.111118    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:56:30.261834 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:30.261799    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:56:30.298576 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:30.298547    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:56:38.452949 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.452911    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"]
Sep 10 15:56:38.455445 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.455423    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:38.457567 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.457540    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-0ce14d90-kserve-self-signed-certs\""
Sep 10 15:56:38.462956 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.462550    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"]
Sep 10 15:56:38.470483 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.470453    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6f6m\" (UniqueName: \"kubernetes.io/projected/68c99855-df37-4751-9752-a996c9963edc-kube-api-access-g6f6m\") pod \"e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") " pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:38.470592 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.470499    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-home\") pod \"e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") " pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:38.470592 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.470521    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-dshm\") pod \"e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") " pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:38.470592 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.470556    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c99855-df37-4751-9752-a996c9963edc-tls-certs\") pod \"e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") " pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:38.470708 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.470606    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-model-cache\") pod \"e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") " pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:38.470708 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.470647    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-tmp-dir\") pod \"e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") " pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:38.571484 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.571441    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-tmp-dir\") pod \"e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") " pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:38.571693 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.571548    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6f6m\" (UniqueName: \"kubernetes.io/projected/68c99855-df37-4751-9752-a996c9963edc-kube-api-access-g6f6m\") pod \"e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") " pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:38.571693 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.571589    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-home\") pod \"e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") " pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:38.571693 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.571611    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-dshm\") pod \"e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") " pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:38.571693 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.571626    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c99855-df37-4751-9752-a996c9963edc-tls-certs\") pod \"e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") " pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:38.571693 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.571666    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-model-cache\") pod \"e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") " pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:38.571970 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.571948    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-tmp-dir\") pod \"e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") " pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:38.572044 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.572022    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-home\") pod \"e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") " pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:38.572081 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.572043    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-model-cache\") pod \"e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") " pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:38.573775 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.573756    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-dshm\") pod \"e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") " pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:38.574185 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.574164    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c99855-df37-4751-9752-a996c9963edc-tls-certs\") pod \"e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") " pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:38.579752 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.579728    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6f6m\" (UniqueName: \"kubernetes.io/projected/68c99855-df37-4751-9752-a996c9963edc-kube-api-access-g6f6m\") pod \"e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") " pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:38.765439 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:38.765250    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:39.037186 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:39.037159    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"]
Sep 10 15:56:39.038819 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:56:39.038783    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68c99855_df37_4751_9752_a996c9963edc.slice/crio-8eac286c7281b75f42ff7febeb6d079f91685853d520570608102a5875c02024 WatchSource:0}: Error finding container 8eac286c7281b75f42ff7febeb6d079f91685853d520570608102a5875c02024: Status 404 returned error can't find the container with id 8eac286c7281b75f42ff7febeb6d079f91685853d520570608102a5875c02024
Sep 10 15:56:39.348961 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:39.348925    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr" event={"ID":"68c99855-df37-4751-9752-a996c9963edc","Type":"ContainerStarted","Data":"e50a258000322161ff953e93d4d6136c3fac8e5bf7942293bdba17e9e3e5a4c5"}
Sep 10 15:56:39.349152 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:39.348976    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr" event={"ID":"68c99855-df37-4751-9752-a996c9963edc","Type":"ContainerStarted","Data":"8eac286c7281b75f42ff7febeb6d079f91685853d520570608102a5875c02024"}
Sep 10 15:56:39.367510 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:39.367456    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr" podStartSLOduration=1.3674392850000001 podStartE2EDuration="1.367439285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:39.363947901 +0000 UTC m=+872.036358082" watchObservedRunningTime="2026-09-10 15:56:39.367439285 +0000 UTC m=+872.039849405"
Sep 10 15:56:42.286019 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:42.285976    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"]
Sep 10 15:56:42.286557 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:42.286243    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc" podUID="e61dfee7-c61d-4bde-a32c-746abe930ac7" containerName="istio-proxy" containerID="cri-o://3ef505f1d2b71f3b3a4c1efb6cffe1db66cf3146ea04155b9d6f59873a1ab2e1" gracePeriod=30
Sep 10 15:56:45.300810 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:45.300769    2572 patch_prober.go:28] interesting pod/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 15:56:45.301193 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:45.300830    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc" podUID="e61dfee7-c61d-4bde-a32c-746abe930ac7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 15:56:47.646716 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.646688    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:56:47.763280 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.763180    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-envoy\") pod \"e61dfee7-c61d-4bde-a32c-746abe930ac7\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") "
Sep 10 15:56:47.763280 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.763226    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-credential-socket\") pod \"e61dfee7-c61d-4bde-a32c-746abe930ac7\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") "
Sep 10 15:56:47.763280 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.763279    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shg7r\" (UniqueName: \"kubernetes.io/projected/e61dfee7-c61d-4bde-a32c-746abe930ac7-kube-api-access-shg7r\") pod \"e61dfee7-c61d-4bde-a32c-746abe930ac7\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") "
Sep 10 15:56:47.763579 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.763342    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-workload-socket\") pod \"e61dfee7-c61d-4bde-a32c-746abe930ac7\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") "
Sep 10 15:56:47.763579 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.763407    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61dfee7-c61d-4bde-a32c-746abe930ac7-istiod-ca-cert\") pod \"e61dfee7-c61d-4bde-a32c-746abe930ac7\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") "
Sep 10 15:56:47.763579 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.763433    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-workload-certs\") pod \"e61dfee7-c61d-4bde-a32c-746abe930ac7\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") "
Sep 10 15:56:47.763579 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.763469    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-token\") pod \"e61dfee7-c61d-4bde-a32c-746abe930ac7\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") "
Sep 10 15:56:47.763579 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.763527    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-data\") pod \"e61dfee7-c61d-4bde-a32c-746abe930ac7\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") "
Sep 10 15:56:47.763579 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.763560    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-podinfo\") pod \"e61dfee7-c61d-4bde-a32c-746abe930ac7\" (UID: \"e61dfee7-c61d-4bde-a32c-746abe930ac7\") "
Sep 10 15:56:47.763887 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.763645    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e61dfee7-c61d-4bde-a32c-746abe930ac7" (UID: "e61dfee7-c61d-4bde-a32c-746abe930ac7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:56:47.763887 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.763698    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e61dfee7-c61d-4bde-a32c-746abe930ac7" (UID: "e61dfee7-c61d-4bde-a32c-746abe930ac7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:56:47.763887 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.763786    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61dfee7-c61d-4bde-a32c-746abe930ac7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e61dfee7-c61d-4bde-a32c-746abe930ac7" (UID: "e61dfee7-c61d-4bde-a32c-746abe930ac7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:56:47.764047 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.763974    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-credential-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:56:47.764047 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.763995    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-workload-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:56:47.764047 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.764009    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e61dfee7-c61d-4bde-a32c-746abe930ac7-istiod-ca-cert\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:56:47.764200 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.764045    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-data" (OuterVolumeSpecName: "istio-data") pod "e61dfee7-c61d-4bde-a32c-746abe930ac7" (UID: "e61dfee7-c61d-4bde-a32c-746abe930ac7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:56:47.764256 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.764206    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e61dfee7-c61d-4bde-a32c-746abe930ac7" (UID: "e61dfee7-c61d-4bde-a32c-746abe930ac7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:56:47.765542 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.765516    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e61dfee7-c61d-4bde-a32c-746abe930ac7" (UID: "e61dfee7-c61d-4bde-a32c-746abe930ac7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:56:47.765788 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.765770    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61dfee7-c61d-4bde-a32c-746abe930ac7-kube-api-access-shg7r" (OuterVolumeSpecName: "kube-api-access-shg7r") pod "e61dfee7-c61d-4bde-a32c-746abe930ac7" (UID: "e61dfee7-c61d-4bde-a32c-746abe930ac7"). InnerVolumeSpecName "kube-api-access-shg7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:56:47.766065 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.766045    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-token" (OuterVolumeSpecName: "istio-token") pod "e61dfee7-c61d-4bde-a32c-746abe930ac7" (UID: "e61dfee7-c61d-4bde-a32c-746abe930ac7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:56:47.766065 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.766060    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e61dfee7-c61d-4bde-a32c-746abe930ac7" (UID: "e61dfee7-c61d-4bde-a32c-746abe930ac7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:56:47.865458 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.865418    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-podinfo\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:56:47.865458 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.865458    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-envoy\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:56:47.865720 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.865472    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shg7r\" (UniqueName: \"kubernetes.io/projected/e61dfee7-c61d-4bde-a32c-746abe930ac7-kube-api-access-shg7r\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:56:47.865720 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.865487    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-workload-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:56:47.865720 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.865501    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-token\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:56:47.865720 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:47.865513    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e61dfee7-c61d-4bde-a32c-746abe930ac7-istio-data\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:56:48.400748 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:48.400709    2572 generic.go:356] "Generic (PLEG): container finished" podID="e61dfee7-c61d-4bde-a32c-746abe930ac7" containerID="3ef505f1d2b71f3b3a4c1efb6cffe1db66cf3146ea04155b9d6f59873a1ab2e1" exitCode=0
Sep 10 15:56:48.400956 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:48.400780    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"
Sep 10 15:56:48.400956 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:48.400790    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc" event={"ID":"e61dfee7-c61d-4bde-a32c-746abe930ac7","Type":"ContainerDied","Data":"3ef505f1d2b71f3b3a4c1efb6cffe1db66cf3146ea04155b9d6f59873a1ab2e1"}
Sep 10 15:56:48.400956 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:48.400872    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc" event={"ID":"e61dfee7-c61d-4bde-a32c-746abe930ac7","Type":"ContainerDied","Data":"a5ec6160906456377a5aabaa8019df8b1c7808d7160edb568352d34ad5523c9a"}
Sep 10 15:56:48.400956 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:48.400895    2572 scope.go:117] "RemoveContainer" containerID="3ef505f1d2b71f3b3a4c1efb6cffe1db66cf3146ea04155b9d6f59873a1ab2e1"
Sep 10 15:56:48.410392 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:48.410369    2572 scope.go:117] "RemoveContainer" containerID="047cc06f895830d8d64c08686838a7a4a9f283b8cb220f57e5bca8e86f6bc5f9"
Sep 10 15:56:48.419544 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:48.419510    2572 scope.go:117] "RemoveContainer" containerID="3ef505f1d2b71f3b3a4c1efb6cffe1db66cf3146ea04155b9d6f59873a1ab2e1"
Sep 10 15:56:48.419728 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:48.419709    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"]
Sep 10 15:56:48.419903 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:56:48.419883    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ef505f1d2b71f3b3a4c1efb6cffe1db66cf3146ea04155b9d6f59873a1ab2e1\": container with ID starting with 3ef505f1d2b71f3b3a4c1efb6cffe1db66cf3146ea04155b9d6f59873a1ab2e1 not found: ID does not exist" containerID="3ef505f1d2b71f3b3a4c1efb6cffe1db66cf3146ea04155b9d6f59873a1ab2e1"
Sep 10 15:56:48.419970 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:48.419916    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ef505f1d2b71f3b3a4c1efb6cffe1db66cf3146ea04155b9d6f59873a1ab2e1"} err="failed to get container status \"3ef505f1d2b71f3b3a4c1efb6cffe1db66cf3146ea04155b9d6f59873a1ab2e1\": rpc error: code = NotFound desc = could not find container \"3ef505f1d2b71f3b3a4c1efb6cffe1db66cf3146ea04155b9d6f59873a1ab2e1\": container with ID starting with 3ef505f1d2b71f3b3a4c1efb6cffe1db66cf3146ea04155b9d6f59873a1ab2e1 not found: ID does not exist"
Sep 10 15:56:48.420033 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:48.419978    2572 scope.go:117] "RemoveContainer" containerID="047cc06f895830d8d64c08686838a7a4a9f283b8cb220f57e5bca8e86f6bc5f9"
Sep 10 15:56:48.420332 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:56:48.420287    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"047cc06f895830d8d64c08686838a7a4a9f283b8cb220f57e5bca8e86f6bc5f9\": container with ID starting with 047cc06f895830d8d64c08686838a7a4a9f283b8cb220f57e5bca8e86f6bc5f9 not found: ID does not exist" containerID="047cc06f895830d8d64c08686838a7a4a9f283b8cb220f57e5bca8e86f6bc5f9"
Sep 10 15:56:48.420415 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:48.420338    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"047cc06f895830d8d64c08686838a7a4a9f283b8cb220f57e5bca8e86f6bc5f9"} err="failed to get container status \"047cc06f895830d8d64c08686838a7a4a9f283b8cb220f57e5bca8e86f6bc5f9\": rpc error: code = NotFound desc = could not find container \"047cc06f895830d8d64c08686838a7a4a9f283b8cb220f57e5bca8e86f6bc5f9\": container with ID starting with 047cc06f895830d8d64c08686838a7a4a9f283b8cb220f57e5bca8e86f6bc5f9 not found: ID does not exist"
Sep 10 15:56:48.424708 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:48.424680    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-0fcd7b0f-gw-openshift-default-596f59bc64-886nc"]
Sep 10 15:56:48.765840 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:48.765739    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:48.765840 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:48.765784    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:48.778281 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:48.778248    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:49.419893 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:49.419862    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:56:49.992706 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:49.992675    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e61dfee7-c61d-4bde-a32c-746abe930ac7" path="/var/lib/kubelet/pods/e61dfee7-c61d-4bde-a32c-746abe930ac7/volumes"
Sep 10 15:56:58.217518 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:56:58.217484    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"]
Sep 10 15:57:07.840204 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:07.840174    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7sm72_8ddd4dc1-1122-41e0-a54a-67fb8008d7b9/ovn-acl-logging/0.log"
Sep 10 15:57:07.844985 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:07.844958    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7sm72_8ddd4dc1-1122-41e0-a54a-67fb8008d7b9/ovn-acl-logging/0.log"
Sep 10 15:57:10.658594 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.658554    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"]
Sep 10 15:57:10.658994 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.658924    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e61dfee7-c61d-4bde-a32c-746abe930ac7" containerName="istio-proxy"
Sep 10 15:57:10.658994 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.658935    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e61dfee7-c61d-4bde-a32c-746abe930ac7" containerName="istio-proxy"
Sep 10 15:57:10.658994 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.658946    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e61dfee7-c61d-4bde-a32c-746abe930ac7" containerName="istio-proxy"
Sep 10 15:57:10.658994 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.658952    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e61dfee7-c61d-4bde-a32c-746abe930ac7" containerName="istio-proxy"
Sep 10 15:57:10.659141 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.659008    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e61dfee7-c61d-4bde-a32c-746abe930ac7" containerName="istio-proxy"
Sep 10 15:57:10.659141 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.659016    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e61dfee7-c61d-4bde-a32c-746abe930ac7" containerName="istio-proxy"
Sep 10 15:57:10.662292 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.662264    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:10.665759 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.665728    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-789be7\"/\"openshift-service-ca.crt\""
Sep 10 15:57:10.665932 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.665763    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-789be7\"/\"kube-root-ca.crt\""
Sep 10 15:57:10.665932 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.665763    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-789be7\"/\"default-dockercfg-d8jk9\""
Sep 10 15:57:10.665932 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.665782    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-789be7\"/\"test-model-789be7-kserve-self-signed-certs\""
Sep 10 15:57:10.675127 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.675092    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"]
Sep 10 15:57:10.803729 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.803683    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-home\") pod \"test-model-789be7-kserve-77574677d9-pf7g5\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") " pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:10.803729 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.803732    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-tls-certs\") pod \"test-model-789be7-kserve-77574677d9-pf7g5\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") " pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:10.803963 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.803798    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-model-cache\") pod \"test-model-789be7-kserve-77574677d9-pf7g5\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") " pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:10.803963 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.803847    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-tmp-dir\") pod \"test-model-789be7-kserve-77574677d9-pf7g5\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") " pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:10.804032 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.803952    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-dshm\") pod \"test-model-789be7-kserve-77574677d9-pf7g5\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") " pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:10.804032 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.804005    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k25pd\" (UniqueName: \"kubernetes.io/projected/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-kube-api-access-k25pd\") pod \"test-model-789be7-kserve-77574677d9-pf7g5\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") " pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:10.905439 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.905397    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-model-cache\") pod \"test-model-789be7-kserve-77574677d9-pf7g5\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") " pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:10.905661 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.905452    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-tmp-dir\") pod \"test-model-789be7-kserve-77574677d9-pf7g5\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") " pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:10.905661 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.905525    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-dshm\") pod \"test-model-789be7-kserve-77574677d9-pf7g5\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") " pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:10.905661 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.905564    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k25pd\" (UniqueName: \"kubernetes.io/projected/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-kube-api-access-k25pd\") pod \"test-model-789be7-kserve-77574677d9-pf7g5\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") " pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:10.905661 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.905618    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-home\") pod \"test-model-789be7-kserve-77574677d9-pf7g5\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") " pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:10.905661 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.905646    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-tls-certs\") pod \"test-model-789be7-kserve-77574677d9-pf7g5\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") " pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:10.905939 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.905843    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-model-cache\") pod \"test-model-789be7-kserve-77574677d9-pf7g5\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") " pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:10.905939 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.905868    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-tmp-dir\") pod \"test-model-789be7-kserve-77574677d9-pf7g5\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") " pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:10.906051 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.906025    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-home\") pod \"test-model-789be7-kserve-77574677d9-pf7g5\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") " pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:10.907798 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.907774    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-dshm\") pod \"test-model-789be7-kserve-77574677d9-pf7g5\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") " pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:10.908387 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.908367    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-tls-certs\") pod \"test-model-789be7-kserve-77574677d9-pf7g5\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") " pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:10.913570 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.913517    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k25pd\" (UniqueName: \"kubernetes.io/projected/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-kube-api-access-k25pd\") pod \"test-model-789be7-kserve-77574677d9-pf7g5\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") " pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:10.972429 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:10.972390    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:11.252777 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:11.252746    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"]
Sep 10 15:57:11.254362 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:57:11.254326    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fd169cb_2e7f_4bfb_b542_3b7e086b7b9a.slice/crio-474cd2e59cee928c9b30960b7651d469e819b68148f67e18f466b64844f7f224 WatchSource:0}: Error finding container 474cd2e59cee928c9b30960b7651d469e819b68148f67e18f466b64844f7f224: Status 404 returned error can't find the container with id 474cd2e59cee928c9b30960b7651d469e819b68148f67e18f466b64844f7f224
Sep 10 15:57:11.540715 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:11.540610    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5" event={"ID":"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a","Type":"ContainerStarted","Data":"47717528425b2250763b4bff9aeaf9d25b1399ad5d545bef93b0d373cd53bc0e"}
Sep 10 15:57:11.540715 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:11.540657    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5" event={"ID":"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a","Type":"ContainerStarted","Data":"474cd2e59cee928c9b30960b7651d469e819b68148f67e18f466b64844f7f224"}
Sep 10 15:57:11.557626 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:11.557569    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5" podStartSLOduration=1.5575513669999999 podStartE2EDuration="1.557551367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:57:11.55682411 +0000 UTC m=+904.229234193" watchObservedRunningTime="2026-09-10 15:57:11.557551367 +0000 UTC m=+904.229961472"
Sep 10 15:57:13.237505 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.237456    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj" podUID="81add4b2-a2e8-4729-8c3d-96d3b1421539" containerName="main" containerID="cri-o://9b8d292592410b7568890496531dc77043c8087d9e7b997e0b207e48122829d6" gracePeriod=45
Sep 10 15:57:13.490886 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.490814    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:57:13.533365 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.533324    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-dshm\") pod \"81add4b2-a2e8-4729-8c3d-96d3b1421539\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") "
Sep 10 15:57:13.533632 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.533412    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4mh8\" (UniqueName: \"kubernetes.io/projected/81add4b2-a2e8-4729-8c3d-96d3b1421539-kube-api-access-l4mh8\") pod \"81add4b2-a2e8-4729-8c3d-96d3b1421539\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") "
Sep 10 15:57:13.533632 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.533518    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-tmp-dir\") pod \"81add4b2-a2e8-4729-8c3d-96d3b1421539\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") "
Sep 10 15:57:13.533632 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.533543    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-model-cache\") pod \"81add4b2-a2e8-4729-8c3d-96d3b1421539\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") "
Sep 10 15:57:13.533632 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.533569    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-home\") pod \"81add4b2-a2e8-4729-8c3d-96d3b1421539\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") "
Sep 10 15:57:13.533632 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.533600    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81add4b2-a2e8-4729-8c3d-96d3b1421539-tls-certs\") pod \"81add4b2-a2e8-4729-8c3d-96d3b1421539\" (UID: \"81add4b2-a2e8-4729-8c3d-96d3b1421539\") "
Sep 10 15:57:13.534515 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.534483    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-model-cache" (OuterVolumeSpecName: "model-cache") pod "81add4b2-a2e8-4729-8c3d-96d3b1421539" (UID: "81add4b2-a2e8-4729-8c3d-96d3b1421539"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:13.534655 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.534520    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81add4b2-a2e8-4729-8c3d-96d3b1421539" (UID: "81add4b2-a2e8-4729-8c3d-96d3b1421539"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:13.535157 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.535125    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-home" (OuterVolumeSpecName: "home") pod "81add4b2-a2e8-4729-8c3d-96d3b1421539" (UID: "81add4b2-a2e8-4729-8c3d-96d3b1421539"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:13.536907 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.536877    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-dshm" (OuterVolumeSpecName: "dshm") pod "81add4b2-a2e8-4729-8c3d-96d3b1421539" (UID: "81add4b2-a2e8-4729-8c3d-96d3b1421539"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:13.537447 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.537412    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81add4b2-a2e8-4729-8c3d-96d3b1421539-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81add4b2-a2e8-4729-8c3d-96d3b1421539" (UID: "81add4b2-a2e8-4729-8c3d-96d3b1421539"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:57:13.537551 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.537476    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81add4b2-a2e8-4729-8c3d-96d3b1421539-kube-api-access-l4mh8" (OuterVolumeSpecName: "kube-api-access-l4mh8") pod "81add4b2-a2e8-4729-8c3d-96d3b1421539" (UID: "81add4b2-a2e8-4729-8c3d-96d3b1421539"). InnerVolumeSpecName "kube-api-access-l4mh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:57:13.554501 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.554464    2572 generic.go:356] "Generic (PLEG): container finished" podID="81add4b2-a2e8-4729-8c3d-96d3b1421539" containerID="9b8d292592410b7568890496531dc77043c8087d9e7b997e0b207e48122829d6" exitCode=0
Sep 10 15:57:13.554673 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.554523    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"
Sep 10 15:57:13.554673 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.554554    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj" event={"ID":"81add4b2-a2e8-4729-8c3d-96d3b1421539","Type":"ContainerDied","Data":"9b8d292592410b7568890496531dc77043c8087d9e7b997e0b207e48122829d6"}
Sep 10 15:57:13.554673 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.554603    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj" event={"ID":"81add4b2-a2e8-4729-8c3d-96d3b1421539","Type":"ContainerDied","Data":"c66d8474b5e2b277bf4b45dc01cc62696878575b2b5a92468e2d3ba6bcf3c440"}
Sep 10 15:57:13.554673 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.554627    2572 scope.go:117] "RemoveContainer" containerID="9b8d292592410b7568890496531dc77043c8087d9e7b997e0b207e48122829d6"
Sep 10 15:57:13.566858 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.566072    2572 scope.go:117] "RemoveContainer" containerID="9b8d292592410b7568890496531dc77043c8087d9e7b997e0b207e48122829d6"
Sep 10 15:57:13.567237 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:57:13.567207    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b8d292592410b7568890496531dc77043c8087d9e7b997e0b207e48122829d6\": container with ID starting with 9b8d292592410b7568890496531dc77043c8087d9e7b997e0b207e48122829d6 not found: ID does not exist" containerID="9b8d292592410b7568890496531dc77043c8087d9e7b997e0b207e48122829d6"
Sep 10 15:57:13.567341 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.567248    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b8d292592410b7568890496531dc77043c8087d9e7b997e0b207e48122829d6"} err="failed to get container status \"9b8d292592410b7568890496531dc77043c8087d9e7b997e0b207e48122829d6\": rpc error: code = NotFound desc = could not find container \"9b8d292592410b7568890496531dc77043c8087d9e7b997e0b207e48122829d6\": container with ID starting with 9b8d292592410b7568890496531dc77043c8087d9e7b997e0b207e48122829d6 not found: ID does not exist"
Sep 10 15:57:13.576154 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.576123    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"]
Sep 10 15:57:13.581873 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.581848    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-314824/autores-model-314824-kserve-779bccc899-hzwbj"]
Sep 10 15:57:13.635134 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.635095    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-dshm\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:13.635134 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.635127    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4mh8\" (UniqueName: \"kubernetes.io/projected/81add4b2-a2e8-4729-8c3d-96d3b1421539-kube-api-access-l4mh8\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:13.635134 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.635139    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-tmp-dir\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:13.635420 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.635151    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-model-cache\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:13.635420 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.635159    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81add4b2-a2e8-4729-8c3d-96d3b1421539-home\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:13.635420 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.635167    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81add4b2-a2e8-4729-8c3d-96d3b1421539-tls-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:13.994095 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:13.994060    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81add4b2-a2e8-4729-8c3d-96d3b1421539" path="/var/lib/kubelet/pods/81add4b2-a2e8-4729-8c3d-96d3b1421539/volumes"
Sep 10 15:57:14.252083 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:14.251993    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"]
Sep 10 15:57:15.261252 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:15.261216    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"]
Sep 10 15:57:20.972879 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:20.972833    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:20.973379 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:20.972894    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:20.985597 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:20.985569    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:21.614279 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:21.614242    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:57:29.272286 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.272245    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx" podUID="77548a93-6c9a-4e79-8751-953f66737362" containerName="main" containerID="cri-o://553bc30f2f9044181a2b1d4d3e173cf46360653f5a0ea6f2322b6c7092c6273a" gracePeriod=45
Sep 10 15:57:29.528343 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.528257    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:57:29.598756 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.598710    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-dshm\") pod \"77548a93-6c9a-4e79-8751-953f66737362\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") "
Sep 10 15:57:29.598936 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.598836    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gv2pc\" (UniqueName: \"kubernetes.io/projected/77548a93-6c9a-4e79-8751-953f66737362-kube-api-access-gv2pc\") pod \"77548a93-6c9a-4e79-8751-953f66737362\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") "
Sep 10 15:57:29.598936 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.598893    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-home\") pod \"77548a93-6c9a-4e79-8751-953f66737362\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") "
Sep 10 15:57:29.598936 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.598929    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-model-cache\") pod \"77548a93-6c9a-4e79-8751-953f66737362\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") "
Sep 10 15:57:29.599087 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.598958    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77548a93-6c9a-4e79-8751-953f66737362-tls-certs\") pod \"77548a93-6c9a-4e79-8751-953f66737362\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") "
Sep 10 15:57:29.599087 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.599018    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-tmp-dir\") pod \"77548a93-6c9a-4e79-8751-953f66737362\" (UID: \"77548a93-6c9a-4e79-8751-953f66737362\") "
Sep 10 15:57:29.599260 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.599225    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-home" (OuterVolumeSpecName: "home") pod "77548a93-6c9a-4e79-8751-953f66737362" (UID: "77548a93-6c9a-4e79-8751-953f66737362"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:29.599398 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.599283    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-model-cache" (OuterVolumeSpecName: "model-cache") pod "77548a93-6c9a-4e79-8751-953f66737362" (UID: "77548a93-6c9a-4e79-8751-953f66737362"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:29.599398 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.599344    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77548a93-6c9a-4e79-8751-953f66737362" (UID: "77548a93-6c9a-4e79-8751-953f66737362"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:29.599516 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.599434    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-home\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:29.599516 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.599454    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-model-cache\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:29.599516 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.599494    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-tmp-dir\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:29.601117 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.601091    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-dshm" (OuterVolumeSpecName: "dshm") pod "77548a93-6c9a-4e79-8751-953f66737362" (UID: "77548a93-6c9a-4e79-8751-953f66737362"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:29.601227 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.601161    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77548a93-6c9a-4e79-8751-953f66737362-kube-api-access-gv2pc" (OuterVolumeSpecName: "kube-api-access-gv2pc") pod "77548a93-6c9a-4e79-8751-953f66737362" (UID: "77548a93-6c9a-4e79-8751-953f66737362"). InnerVolumeSpecName "kube-api-access-gv2pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:57:29.601227 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.601171    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77548a93-6c9a-4e79-8751-953f66737362-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77548a93-6c9a-4e79-8751-953f66737362" (UID: "77548a93-6c9a-4e79-8751-953f66737362"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:57:29.651062 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.651021    2572 generic.go:356] "Generic (PLEG): container finished" podID="77548a93-6c9a-4e79-8751-953f66737362" containerID="553bc30f2f9044181a2b1d4d3e173cf46360653f5a0ea6f2322b6c7092c6273a" exitCode=0
Sep 10 15:57:29.651244 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.651070    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"
Sep 10 15:57:29.651244 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.651111    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx" event={"ID":"77548a93-6c9a-4e79-8751-953f66737362","Type":"ContainerDied","Data":"553bc30f2f9044181a2b1d4d3e173cf46360653f5a0ea6f2322b6c7092c6273a"}
Sep 10 15:57:29.651244 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.651157    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx" event={"ID":"77548a93-6c9a-4e79-8751-953f66737362","Type":"ContainerDied","Data":"240afee1b95fb3e14ab5b4dede9b381bb8db49e19fec8f33b21553ee106ba06c"}
Sep 10 15:57:29.651244 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.651176    2572 scope.go:117] "RemoveContainer" containerID="553bc30f2f9044181a2b1d4d3e173cf46360653f5a0ea6f2322b6c7092c6273a"
Sep 10 15:57:29.661028 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.661008    2572 scope.go:117] "RemoveContainer" containerID="553bc30f2f9044181a2b1d4d3e173cf46360653f5a0ea6f2322b6c7092c6273a"
Sep 10 15:57:29.661363 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:57:29.661342    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"553bc30f2f9044181a2b1d4d3e173cf46360653f5a0ea6f2322b6c7092c6273a\": container with ID starting with 553bc30f2f9044181a2b1d4d3e173cf46360653f5a0ea6f2322b6c7092c6273a not found: ID does not exist" containerID="553bc30f2f9044181a2b1d4d3e173cf46360653f5a0ea6f2322b6c7092c6273a"
Sep 10 15:57:29.661445 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.661373    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553bc30f2f9044181a2b1d4d3e173cf46360653f5a0ea6f2322b6c7092c6273a"} err="failed to get container status \"553bc30f2f9044181a2b1d4d3e173cf46360653f5a0ea6f2322b6c7092c6273a\": rpc error: code = NotFound desc = could not find container \"553bc30f2f9044181a2b1d4d3e173cf46360653f5a0ea6f2322b6c7092c6273a\": container with ID starting with 553bc30f2f9044181a2b1d4d3e173cf46360653f5a0ea6f2322b6c7092c6273a not found: ID does not exist"
Sep 10 15:57:29.672639 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.672611    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"]
Sep 10 15:57:29.676010 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.675977    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-0ce14d90-kserve-778dcdc95f-gqbzx"]
Sep 10 15:57:29.700849 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.700812    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77548a93-6c9a-4e79-8751-953f66737362-dshm\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:29.700849 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.700845    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv2pc\" (UniqueName: \"kubernetes.io/projected/77548a93-6c9a-4e79-8751-953f66737362-kube-api-access-gv2pc\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:29.701029 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.700860    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77548a93-6c9a-4e79-8751-953f66737362-tls-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:29.992714 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:29.992680    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77548a93-6c9a-4e79-8751-953f66737362" path="/var/lib/kubelet/pods/77548a93-6c9a-4e79-8751-953f66737362/volumes"
Sep 10 15:57:30.281626 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.281530    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr" podUID="68c99855-df37-4751-9752-a996c9963edc" containerName="main" containerID="cri-o://e50a258000322161ff953e93d4d6136c3fac8e5bf7942293bdba17e9e3e5a4c5" gracePeriod=45
Sep 10 15:57:30.548249 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.548220    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:57:30.659160 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.659118    2572 generic.go:356] "Generic (PLEG): container finished" podID="68c99855-df37-4751-9752-a996c9963edc" containerID="e50a258000322161ff953e93d4d6136c3fac8e5bf7942293bdba17e9e3e5a4c5" exitCode=0
Sep 10 15:57:30.659160 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.659166    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"
Sep 10 15:57:30.659441 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.659207    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr" event={"ID":"68c99855-df37-4751-9752-a996c9963edc","Type":"ContainerDied","Data":"e50a258000322161ff953e93d4d6136c3fac8e5bf7942293bdba17e9e3e5a4c5"}
Sep 10 15:57:30.659441 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.659254    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr" event={"ID":"68c99855-df37-4751-9752-a996c9963edc","Type":"ContainerDied","Data":"8eac286c7281b75f42ff7febeb6d079f91685853d520570608102a5875c02024"}
Sep 10 15:57:30.659441 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.659279    2572 scope.go:117] "RemoveContainer" containerID="e50a258000322161ff953e93d4d6136c3fac8e5bf7942293bdba17e9e3e5a4c5"
Sep 10 15:57:30.668248 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.668226    2572 scope.go:117] "RemoveContainer" containerID="e50a258000322161ff953e93d4d6136c3fac8e5bf7942293bdba17e9e3e5a4c5"
Sep 10 15:57:30.668636 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:57:30.668615    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50a258000322161ff953e93d4d6136c3fac8e5bf7942293bdba17e9e3e5a4c5\": container with ID starting with e50a258000322161ff953e93d4d6136c3fac8e5bf7942293bdba17e9e3e5a4c5 not found: ID does not exist" containerID="e50a258000322161ff953e93d4d6136c3fac8e5bf7942293bdba17e9e3e5a4c5"
Sep 10 15:57:30.668697 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.668645    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50a258000322161ff953e93d4d6136c3fac8e5bf7942293bdba17e9e3e5a4c5"} err="failed to get container status \"e50a258000322161ff953e93d4d6136c3fac8e5bf7942293bdba17e9e3e5a4c5\": rpc error: code = NotFound desc = could not find container \"e50a258000322161ff953e93d4d6136c3fac8e5bf7942293bdba17e9e3e5a4c5\": container with ID starting with e50a258000322161ff953e93d4d6136c3fac8e5bf7942293bdba17e9e3e5a4c5 not found: ID does not exist"
Sep 10 15:57:30.710258 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.710225    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-tmp-dir\") pod \"68c99855-df37-4751-9752-a996c9963edc\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") "
Sep 10 15:57:30.710458 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.710347    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-dshm\") pod \"68c99855-df37-4751-9752-a996c9963edc\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") "
Sep 10 15:57:30.710458 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.710402    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6f6m\" (UniqueName: \"kubernetes.io/projected/68c99855-df37-4751-9752-a996c9963edc-kube-api-access-g6f6m\") pod \"68c99855-df37-4751-9752-a996c9963edc\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") "
Sep 10 15:57:30.710458 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.710425    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c99855-df37-4751-9752-a996c9963edc-tls-certs\") pod \"68c99855-df37-4751-9752-a996c9963edc\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") "
Sep 10 15:57:30.710642 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.710462    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-model-cache\") pod \"68c99855-df37-4751-9752-a996c9963edc\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") "
Sep 10 15:57:30.710642 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.710516    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "68c99855-df37-4751-9752-a996c9963edc" (UID: "68c99855-df37-4751-9752-a996c9963edc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:30.710642 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.710530    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-home\") pod \"68c99855-df37-4751-9752-a996c9963edc\" (UID: \"68c99855-df37-4751-9752-a996c9963edc\") "
Sep 10 15:57:30.710790 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.710764    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-model-cache" (OuterVolumeSpecName: "model-cache") pod "68c99855-df37-4751-9752-a996c9963edc" (UID: "68c99855-df37-4751-9752-a996c9963edc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:30.710861 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.710775    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-home" (OuterVolumeSpecName: "home") pod "68c99855-df37-4751-9752-a996c9963edc" (UID: "68c99855-df37-4751-9752-a996c9963edc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:30.710926 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.710909    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-model-cache\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:30.710926 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.710923    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-home\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:30.711039 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.710931    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-tmp-dir\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:30.712602 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.712576    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c99855-df37-4751-9752-a996c9963edc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68c99855-df37-4751-9752-a996c9963edc" (UID: "68c99855-df37-4751-9752-a996c9963edc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:57:30.712742 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.712602    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-dshm" (OuterVolumeSpecName: "dshm") pod "68c99855-df37-4751-9752-a996c9963edc" (UID: "68c99855-df37-4751-9752-a996c9963edc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:30.712742 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.712698    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c99855-df37-4751-9752-a996c9963edc-kube-api-access-g6f6m" (OuterVolumeSpecName: "kube-api-access-g6f6m") pod "68c99855-df37-4751-9752-a996c9963edc" (UID: "68c99855-df37-4751-9752-a996c9963edc"). InnerVolumeSpecName "kube-api-access-g6f6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:57:30.812391 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.812349    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c99855-df37-4751-9752-a996c9963edc-dshm\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:30.812391 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.812386    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6f6m\" (UniqueName: \"kubernetes.io/projected/68c99855-df37-4751-9752-a996c9963edc-kube-api-access-g6f6m\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:30.812607 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.812403    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c99855-df37-4751-9752-a996c9963edc-tls-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:30.980698 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.980658    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"]
Sep 10 15:57:30.984838 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:30.984809    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-0ce14d90-kserve-759888bdd9-wxzgr"]
Sep 10 15:57:31.992961 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:31.992925    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68c99855-df37-4751-9752-a996c9963edc" path="/var/lib/kubelet/pods/68c99855-df37-4751-9752-a996c9963edc/volumes"
Sep 10 15:57:35.368223 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:35.368189    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"]
Sep 10 15:57:35.368724 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:35.368425    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8" podUID="da1000a4-2fc8-47f2-9423-689b97933a5a" containerName="istio-proxy" containerID="cri-o://2156655f191b50fc4ffe170bce1ca4e5db431b31e579bbcbffa165942765541d" gracePeriod=30
Sep 10 15:57:36.184847 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.184816    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"]
Sep 10 15:57:36.185178 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.185166    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77548a93-6c9a-4e79-8751-953f66737362" containerName="main"
Sep 10 15:57:36.185224 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.185180    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="77548a93-6c9a-4e79-8751-953f66737362" containerName="main"
Sep 10 15:57:36.185224 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.185190    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68c99855-df37-4751-9752-a996c9963edc" containerName="main"
Sep 10 15:57:36.185224 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.185196    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="68c99855-df37-4751-9752-a996c9963edc" containerName="main"
Sep 10 15:57:36.185224 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.185204    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81add4b2-a2e8-4729-8c3d-96d3b1421539" containerName="main"
Sep 10 15:57:36.185224 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.185209    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="81add4b2-a2e8-4729-8c3d-96d3b1421539" containerName="main"
Sep 10 15:57:36.185395 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.185268    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="68c99855-df37-4751-9752-a996c9963edc" containerName="main"
Sep 10 15:57:36.185395 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.185277    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="77548a93-6c9a-4e79-8751-953f66737362" containerName="main"
Sep 10 15:57:36.185395 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.185285    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="81add4b2-a2e8-4729-8c3d-96d3b1421539" containerName="main"
Sep 10 15:57:36.188480 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.188463    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.191618 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.191598    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-5c1d26-openshift-default-dockercfg-6pttb\""
Sep 10 15:57:36.206862 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.206833    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"]
Sep 10 15:57:36.367092 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.367056    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a56ab010-1a58-46ce-82c1-ab7232d408b7-istiod-ca-cert\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.367287 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.367111    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qclww\" (UniqueName: \"kubernetes.io/projected/a56ab010-1a58-46ce-82c1-ab7232d408b7-kube-api-access-qclww\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.367287 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.367134    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-envoy\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.367287 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.367215    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-workload-socket\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.367466 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.367287    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-token\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.367466 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.367335    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-credential-socket\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.367466 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.367363    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-data\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.367466 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.367390    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-podinfo\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.367466 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.367409    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-workload-certs\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.468137 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.468051    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a56ab010-1a58-46ce-82c1-ab7232d408b7-istiod-ca-cert\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.468137 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.468101    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qclww\" (UniqueName: \"kubernetes.io/projected/a56ab010-1a58-46ce-82c1-ab7232d408b7-kube-api-access-qclww\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.468137 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.468125    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-envoy\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.468747 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.468152    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-workload-socket\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.468747 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.468291    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-token\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.468747 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.468364    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-credential-socket\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.468747 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.468392    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-data\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.468747 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.468549    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-podinfo\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.468747 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.468598    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-workload-certs\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.468747 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.468726    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-data\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.469101 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.468762    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-workload-socket\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.469101 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.468780    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-credential-socket\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.469101 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.468890    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a56ab010-1a58-46ce-82c1-ab7232d408b7-istiod-ca-cert\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.469101 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.468941    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-workload-certs\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.470592 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.470567    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-envoy\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.471133 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.471108    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-podinfo\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.476685 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.476657    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-token\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.478130 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.478106    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qclww\" (UniqueName: \"kubernetes.io/projected/a56ab010-1a58-46ce-82c1-ab7232d408b7-kube-api-access-qclww\") pod \"e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") " pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.498976 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.498934    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:36.782815 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.782779    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"]
Sep 10 15:57:36.784462 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:57:36.784435    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda56ab010_1a58_46ce_82c1_ab7232d408b7.slice/crio-bd0845fc430287baf407be1bd5a3e9af801fb970392b0a7ea539548ac616f15a WatchSource:0}: Error finding container bd0845fc430287baf407be1bd5a3e9af801fb970392b0a7ea539548ac616f15a: Status 404 returned error can't find the container with id bd0845fc430287baf407be1bd5a3e9af801fb970392b0a7ea539548ac616f15a
Sep 10 15:57:36.787075 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.787043    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:57:36.787161 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.787107    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:57:36.787161 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:36.787138    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:57:37.697831 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:37.697788    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh" event={"ID":"a56ab010-1a58-46ce-82c1-ab7232d408b7","Type":"ContainerStarted","Data":"202c32ea149c1a093ad775236e5023a8a7658835b46d811d28bdd0a3e4fee087"}
Sep 10 15:57:37.698216 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:37.697854    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh" event={"ID":"a56ab010-1a58-46ce-82c1-ab7232d408b7","Type":"ContainerStarted","Data":"bd0845fc430287baf407be1bd5a3e9af801fb970392b0a7ea539548ac616f15a"}
Sep 10 15:57:37.725035 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:37.724976    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh" podStartSLOduration=1.7249590879999999 podStartE2EDuration="1.724959088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:57:37.722043777 +0000 UTC m=+930.394453862" watchObservedRunningTime="2026-09-10 15:57:37.724959088 +0000 UTC m=+930.397369171"
Sep 10 15:57:38.499389 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:38.499347    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:38.505058 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:38.505023    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:38.703659 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:38.703631    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:38.704714 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:38.704696    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:39.512226 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:39.512180    2572 patch_prober.go:28] interesting pod/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 15:57:39.512401 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:39.512249    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8" podUID="da1000a4-2fc8-47f2-9423-689b97933a5a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 15:57:40.616774 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.616743    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:57:40.716635 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.716545    2572 generic.go:356] "Generic (PLEG): container finished" podID="da1000a4-2fc8-47f2-9423-689b97933a5a" containerID="2156655f191b50fc4ffe170bce1ca4e5db431b31e579bbcbffa165942765541d" exitCode=0
Sep 10 15:57:40.716635 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.716594    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"
Sep 10 15:57:40.716831 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.716630    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8" event={"ID":"da1000a4-2fc8-47f2-9423-689b97933a5a","Type":"ContainerDied","Data":"2156655f191b50fc4ffe170bce1ca4e5db431b31e579bbcbffa165942765541d"}
Sep 10 15:57:40.716831 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.716688    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8" event={"ID":"da1000a4-2fc8-47f2-9423-689b97933a5a","Type":"ContainerDied","Data":"3a9b0a3ab4d0fcffada16c651ba900a882ae2781d8c6760864ac3a5cceaa4aed"}
Sep 10 15:57:40.716831 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.716766    2572 scope.go:117] "RemoveContainer" containerID="2156655f191b50fc4ffe170bce1ca4e5db431b31e579bbcbffa165942765541d"
Sep 10 15:57:40.726339 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.726290    2572 scope.go:117] "RemoveContainer" containerID="de626fa1f5323bdf578bd32a3757c0aabdfb58a73ba62df235654db6db609281"
Sep 10 15:57:40.735284 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.735260    2572 scope.go:117] "RemoveContainer" containerID="2156655f191b50fc4ffe170bce1ca4e5db431b31e579bbcbffa165942765541d"
Sep 10 15:57:40.735577 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:57:40.735555    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2156655f191b50fc4ffe170bce1ca4e5db431b31e579bbcbffa165942765541d\": container with ID starting with 2156655f191b50fc4ffe170bce1ca4e5db431b31e579bbcbffa165942765541d not found: ID does not exist" containerID="2156655f191b50fc4ffe170bce1ca4e5db431b31e579bbcbffa165942765541d"
Sep 10 15:57:40.735632 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.735587    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2156655f191b50fc4ffe170bce1ca4e5db431b31e579bbcbffa165942765541d"} err="failed to get container status \"2156655f191b50fc4ffe170bce1ca4e5db431b31e579bbcbffa165942765541d\": rpc error: code = NotFound desc = could not find container \"2156655f191b50fc4ffe170bce1ca4e5db431b31e579bbcbffa165942765541d\": container with ID starting with 2156655f191b50fc4ffe170bce1ca4e5db431b31e579bbcbffa165942765541d not found: ID does not exist"
Sep 10 15:57:40.735632 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.735606    2572 scope.go:117] "RemoveContainer" containerID="de626fa1f5323bdf578bd32a3757c0aabdfb58a73ba62df235654db6db609281"
Sep 10 15:57:40.735845 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:57:40.735825    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de626fa1f5323bdf578bd32a3757c0aabdfb58a73ba62df235654db6db609281\": container with ID starting with de626fa1f5323bdf578bd32a3757c0aabdfb58a73ba62df235654db6db609281 not found: ID does not exist" containerID="de626fa1f5323bdf578bd32a3757c0aabdfb58a73ba62df235654db6db609281"
Sep 10 15:57:40.735887 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.735856    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de626fa1f5323bdf578bd32a3757c0aabdfb58a73ba62df235654db6db609281"} err="failed to get container status \"de626fa1f5323bdf578bd32a3757c0aabdfb58a73ba62df235654db6db609281\": rpc error: code = NotFound desc = could not find container \"de626fa1f5323bdf578bd32a3757c0aabdfb58a73ba62df235654db6db609281\": container with ID starting with de626fa1f5323bdf578bd32a3757c0aabdfb58a73ba62df235654db6db609281 not found: ID does not exist"
Sep 10 15:57:40.808897 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.808853    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-credential-socket\") pod \"da1000a4-2fc8-47f2-9423-689b97933a5a\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") "
Sep 10 15:57:40.808897 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.808905    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-podinfo\") pod \"da1000a4-2fc8-47f2-9423-689b97933a5a\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") "
Sep 10 15:57:40.809161 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.808949    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-data\") pod \"da1000a4-2fc8-47f2-9423-689b97933a5a\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") "
Sep 10 15:57:40.809161 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.808970    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-token\") pod \"da1000a4-2fc8-47f2-9423-689b97933a5a\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") "
Sep 10 15:57:40.809161 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.808996    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-envoy\") pod \"da1000a4-2fc8-47f2-9423-689b97933a5a\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") "
Sep 10 15:57:40.809161 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.809088    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7j9z\" (UniqueName: \"kubernetes.io/projected/da1000a4-2fc8-47f2-9423-689b97933a5a-kube-api-access-k7j9z\") pod \"da1000a4-2fc8-47f2-9423-689b97933a5a\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") "
Sep 10 15:57:40.809161 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.809121    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-workload-certs\") pod \"da1000a4-2fc8-47f2-9423-689b97933a5a\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") "
Sep 10 15:57:40.809460 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.809167    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1000a4-2fc8-47f2-9423-689b97933a5a-istiod-ca-cert\") pod \"da1000a4-2fc8-47f2-9423-689b97933a5a\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") "
Sep 10 15:57:40.809460 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.809195    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-workload-socket\") pod \"da1000a4-2fc8-47f2-9423-689b97933a5a\" (UID: \"da1000a4-2fc8-47f2-9423-689b97933a5a\") "
Sep 10 15:57:40.809460 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.809234    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-data" (OuterVolumeSpecName: "istio-data") pod "da1000a4-2fc8-47f2-9423-689b97933a5a" (UID: "da1000a4-2fc8-47f2-9423-689b97933a5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:40.809460 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.809393    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "da1000a4-2fc8-47f2-9423-689b97933a5a" (UID: "da1000a4-2fc8-47f2-9423-689b97933a5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:40.809668 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.809488    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "da1000a4-2fc8-47f2-9423-689b97933a5a" (UID: "da1000a4-2fc8-47f2-9423-689b97933a5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:40.809668 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.809599    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "da1000a4-2fc8-47f2-9423-689b97933a5a" (UID: "da1000a4-2fc8-47f2-9423-689b97933a5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:40.809668 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.809618    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-data\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:40.809668 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.809638    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-workload-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:40.809668 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.809641    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da1000a4-2fc8-47f2-9423-689b97933a5a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "da1000a4-2fc8-47f2-9423-689b97933a5a" (UID: "da1000a4-2fc8-47f2-9423-689b97933a5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:57:40.809668 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.809654    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-credential-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:40.811446 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.811413    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "da1000a4-2fc8-47f2-9423-689b97933a5a" (UID: "da1000a4-2fc8-47f2-9423-689b97933a5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:40.811532 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.811474    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1000a4-2fc8-47f2-9423-689b97933a5a-kube-api-access-k7j9z" (OuterVolumeSpecName: "kube-api-access-k7j9z") pod "da1000a4-2fc8-47f2-9423-689b97933a5a" (UID: "da1000a4-2fc8-47f2-9423-689b97933a5a"). InnerVolumeSpecName "kube-api-access-k7j9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:57:40.811684 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.811668    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "da1000a4-2fc8-47f2-9423-689b97933a5a" (UID: "da1000a4-2fc8-47f2-9423-689b97933a5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:57:40.811792 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.811772    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-token" (OuterVolumeSpecName: "istio-token") pod "da1000a4-2fc8-47f2-9423-689b97933a5a" (UID: "da1000a4-2fc8-47f2-9423-689b97933a5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:57:40.910362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.910325    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-podinfo\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:40.910362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.910355    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-token\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:40.910362 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.910365    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-istio-envoy\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:40.910708 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.910377    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7j9z\" (UniqueName: \"kubernetes.io/projected/da1000a4-2fc8-47f2-9423-689b97933a5a-kube-api-access-k7j9z\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:40.910708 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.910392    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1000a4-2fc8-47f2-9423-689b97933a5a-istiod-ca-cert\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:40.910708 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:40.910405    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1000a4-2fc8-47f2-9423-689b97933a5a-workload-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:41.038775 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:41.038742    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"]
Sep 10 15:57:41.042134 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:41.042104    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-314824-openshift-default-7fb76f75d6-rfst8"]
Sep 10 15:57:41.991823 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:41.991787    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da1000a4-2fc8-47f2-9423-689b97933a5a" path="/var/lib/kubelet/pods/da1000a4-2fc8-47f2-9423-689b97933a5a/volumes"
Sep 10 15:57:46.857489 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:46.857450    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r"]
Sep 10 15:57:46.857878 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:46.857818    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da1000a4-2fc8-47f2-9423-689b97933a5a" containerName="istio-proxy"
Sep 10 15:57:46.857878 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:46.857830    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="da1000a4-2fc8-47f2-9423-689b97933a5a" containerName="istio-proxy"
Sep 10 15:57:46.857878 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:46.857845    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da1000a4-2fc8-47f2-9423-689b97933a5a" containerName="istio-proxy"
Sep 10 15:57:46.857878 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:46.857851    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="da1000a4-2fc8-47f2-9423-689b97933a5a" containerName="istio-proxy"
Sep 10 15:57:46.858008 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:46.857909    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="da1000a4-2fc8-47f2-9423-689b97933a5a" containerName="istio-proxy"
Sep 10 15:57:46.858008 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:46.857919    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="da1000a4-2fc8-47f2-9423-689b97933a5a" containerName="istio-proxy"
Sep 10 15:57:46.862483 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:46.862459    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r"
Sep 10 15:57:46.865539 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:46.865514    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\""
Sep 10 15:57:46.865678 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:46.865521    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 15:57:46.868105 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:46.868079    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r"]
Sep 10 15:57:46.966895 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:46.966854    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3240d0-6f17-4d1e-a97b-8004f0924d47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r\" (UID: \"9b3240d0-6f17-4d1e-a97b-8004f0924d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r"
Sep 10 15:57:47.067420 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:47.067377    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3240d0-6f17-4d1e-a97b-8004f0924d47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r\" (UID: \"9b3240d0-6f17-4d1e-a97b-8004f0924d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r"
Sep 10 15:57:47.068143 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:47.068121    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3240d0-6f17-4d1e-a97b-8004f0924d47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r\" (UID: \"9b3240d0-6f17-4d1e-a97b-8004f0924d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r"
Sep 10 15:57:47.174961 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:47.174871    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r"
Sep 10 15:57:47.445047 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:47.445019    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r"]
Sep 10 15:57:47.447024 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:57:47.446997    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3240d0_6f17_4d1e_a97b_8004f0924d47.slice/crio-bd2d93cfa08d5bea1b19880ad5695d098a932d51070016d7c6ea9b718f8c1b6b WatchSource:0}: Error finding container bd2d93cfa08d5bea1b19880ad5695d098a932d51070016d7c6ea9b718f8c1b6b: Status 404 returned error can't find the container with id bd2d93cfa08d5bea1b19880ad5695d098a932d51070016d7c6ea9b718f8c1b6b
Sep 10 15:57:47.765369 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:47.765325    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r" event={"ID":"9b3240d0-6f17-4d1e-a97b-8004f0924d47","Type":"ContainerStarted","Data":"b5330048f62d6c686a7928fe05a30af4fdf6679e9a3618af4011bd0e97f27d27"}
Sep 10 15:57:47.765544 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:47.765428    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r" event={"ID":"9b3240d0-6f17-4d1e-a97b-8004f0924d47","Type":"ContainerStarted","Data":"bd2d93cfa08d5bea1b19880ad5695d098a932d51070016d7c6ea9b718f8c1b6b"}
Sep 10 15:57:47.780879 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:47.780824    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r" podStartSLOduration=1.780807746 podStartE2EDuration="1.780807746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:57:47.779042416 +0000 UTC m=+940.451452492" watchObservedRunningTime="2026-09-10 15:57:47.780807746 +0000 UTC m=+940.453217828"
Sep 10 15:57:48.773455 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:48.773419    2572 generic.go:356] "Generic (PLEG): container finished" podID="9b3240d0-6f17-4d1e-a97b-8004f0924d47" containerID="b5330048f62d6c686a7928fe05a30af4fdf6679e9a3618af4011bd0e97f27d27" exitCode=0
Sep 10 15:57:48.773825 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:48.773512    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r" event={"ID":"9b3240d0-6f17-4d1e-a97b-8004f0924d47","Type":"ContainerDied","Data":"b5330048f62d6c686a7928fe05a30af4fdf6679e9a3618af4011bd0e97f27d27"}
Sep 10 15:57:49.952933 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:49.952901    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r"
Sep 10 15:57:49.994795 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:49.994759    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3240d0-6f17-4d1e-a97b-8004f0924d47-maas-api-service-ca\") pod \"9b3240d0-6f17-4d1e-a97b-8004f0924d47\" (UID: \"9b3240d0-6f17-4d1e-a97b-8004f0924d47\") "
Sep 10 15:57:49.995092 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:49.995069    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b3240d0-6f17-4d1e-a97b-8004f0924d47-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9b3240d0-6f17-4d1e-a97b-8004f0924d47" (UID: "9b3240d0-6f17-4d1e-a97b-8004f0924d47"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:57:50.095563 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:50.095518    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3240d0-6f17-4d1e-a97b-8004f0924d47-maas-api-service-ca\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:57:50.786719 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:50.786688    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r"
Sep 10 15:57:50.786904 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:50.786721    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r" event={"ID":"9b3240d0-6f17-4d1e-a97b-8004f0924d47","Type":"ContainerDied","Data":"bd2d93cfa08d5bea1b19880ad5695d098a932d51070016d7c6ea9b718f8c1b6b"}
Sep 10 15:57:50.786904 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:50.786771    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd2d93cfa08d5bea1b19880ad5695d098a932d51070016d7c6ea9b718f8c1b6b"
Sep 10 15:57:53.978474 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:53.978434    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r"]
Sep 10 15:57:53.984381 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:53.984339    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7a3187a1-mfk5r"]
Sep 10 15:57:53.993263 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:53.993229    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b3240d0-6f17-4d1e-a97b-8004f0924d47" path="/var/lib/kubelet/pods/9b3240d0-6f17-4d1e-a97b-8004f0924d47/volumes"
Sep 10 15:57:54.746582 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:54.746493    2572 patch_prober.go:28] interesting pod/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 15:57:54.746741 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:54.746567    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh" podUID="a56ab010-1a58-46ce-82c1-ab7232d408b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 15:57:56.828103 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:56.828066    2572 generic.go:356] "Generic (PLEG): container finished" podID="a56ab010-1a58-46ce-82c1-ab7232d408b7" containerID="202c32ea149c1a093ad775236e5023a8a7658835b46d811d28bdd0a3e4fee087" exitCode=0
Sep 10 15:57:56.828538 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:56.828114    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh" event={"ID":"a56ab010-1a58-46ce-82c1-ab7232d408b7","Type":"ContainerDied","Data":"202c32ea149c1a093ad775236e5023a8a7658835b46d811d28bdd0a3e4fee087"}
Sep 10 15:57:56.828610 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:56.828590    2572 scope.go:117] "RemoveContainer" containerID="202c32ea149c1a093ad775236e5023a8a7658835b46d811d28bdd0a3e4fee087"
Sep 10 15:57:56.829689 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:56.829662    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:57:56.829760 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:56.829732    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:57:56.829797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:56.829765    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:57:57.839981 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:57.839941    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh" event={"ID":"a56ab010-1a58-46ce-82c1-ab7232d408b7","Type":"ContainerStarted","Data":"3c6165fa36a5cfafd0b0a34cde031a57c29cd639e34f3d235f083b584129ed8f"}
Sep 10 15:57:58.499317 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:58.499244    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:57:58.499570 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:58.499548    2572 patch_prober.go:28] interesting pod/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 15:57:58.499625 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:58.499595    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh" podUID="a56ab010-1a58-46ce-82c1-ab7232d408b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 15:57:59.500318 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:59.500278    2572 patch_prober.go:28] interesting pod/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 15:57:59.500811 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:57:59.500358    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh" podUID="a56ab010-1a58-46ce-82c1-ab7232d408b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 15:58:00.499714 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.499669    2572 patch_prober.go:28] interesting pod/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 15:58:00.499985 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.499742    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh" podUID="a56ab010-1a58-46ce-82c1-ab7232d408b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 15:58:00.745888 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.745849    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"]
Sep 10 15:58:00.747108 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.747074    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b3240d0-6f17-4d1e-a97b-8004f0924d47" containerName="revoke-keys"
Sep 10 15:58:00.747263 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.747252    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b3240d0-6f17-4d1e-a97b-8004f0924d47" containerName="revoke-keys"
Sep 10 15:58:00.747519 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.747507    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b3240d0-6f17-4d1e-a97b-8004f0924d47" containerName="revoke-keys"
Sep 10 15:58:00.752323 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.751905    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:00.757708 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.757676    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5d6f34\"/\"kube-root-ca.crt\""
Sep 10 15:58:00.757854 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.757772    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5d6f34\"/\"openshift-service-ca.crt\""
Sep 10 15:58:00.758075 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.758054    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5d6f34\"/\"default-dockercfg-dhx9z\""
Sep 10 15:58:00.758215 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.758194    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5d6f34\"/\"test-model-5d6f34-kserve-self-signed-certs\""
Sep 10 15:58:00.759602 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.759585    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"]
Sep 10 15:58:00.796751 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.796720    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14951fbb-f101-46ca-b3fc-5005137ae41b-tls-certs\") pod \"test-model-5d6f34-kserve-7d5896f6df-b9dtm\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") " pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:00.796923 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.796757    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-home\") pod \"test-model-5d6f34-kserve-7d5896f6df-b9dtm\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") " pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:00.796923 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.796789    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-tmp-dir\") pod \"test-model-5d6f34-kserve-7d5896f6df-b9dtm\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") " pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:00.796923 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.796855    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-dshm\") pod \"test-model-5d6f34-kserve-7d5896f6df-b9dtm\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") " pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:00.796923 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.796888    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-model-cache\") pod \"test-model-5d6f34-kserve-7d5896f6df-b9dtm\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") " pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:00.796923 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.796905    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d94hr\" (UniqueName: \"kubernetes.io/projected/14951fbb-f101-46ca-b3fc-5005137ae41b-kube-api-access-d94hr\") pod \"test-model-5d6f34-kserve-7d5896f6df-b9dtm\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") " pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:00.898011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.897972    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-dshm\") pod \"test-model-5d6f34-kserve-7d5896f6df-b9dtm\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") " pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:00.898011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.898022    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-model-cache\") pod \"test-model-5d6f34-kserve-7d5896f6df-b9dtm\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") " pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:00.898295 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.898143    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d94hr\" (UniqueName: \"kubernetes.io/projected/14951fbb-f101-46ca-b3fc-5005137ae41b-kube-api-access-d94hr\") pod \"test-model-5d6f34-kserve-7d5896f6df-b9dtm\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") " pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:00.898411 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.898291    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14951fbb-f101-46ca-b3fc-5005137ae41b-tls-certs\") pod \"test-model-5d6f34-kserve-7d5896f6df-b9dtm\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") " pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:00.898411 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.898362    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-home\") pod \"test-model-5d6f34-kserve-7d5896f6df-b9dtm\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") " pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:00.898411 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.898405    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-tmp-dir\") pod \"test-model-5d6f34-kserve-7d5896f6df-b9dtm\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") " pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:00.898591 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.898512    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-model-cache\") pod \"test-model-5d6f34-kserve-7d5896f6df-b9dtm\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") " pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:00.898698 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.898673    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-home\") pod \"test-model-5d6f34-kserve-7d5896f6df-b9dtm\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") " pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:00.898785 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.898767    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-tmp-dir\") pod \"test-model-5d6f34-kserve-7d5896f6df-b9dtm\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") " pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:00.900387 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.900365    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-dshm\") pod \"test-model-5d6f34-kserve-7d5896f6df-b9dtm\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") " pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:00.900652 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.900635    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14951fbb-f101-46ca-b3fc-5005137ae41b-tls-certs\") pod \"test-model-5d6f34-kserve-7d5896f6df-b9dtm\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") " pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:00.906584 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:00.906554    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d94hr\" (UniqueName: \"kubernetes.io/projected/14951fbb-f101-46ca-b3fc-5005137ae41b-kube-api-access-d94hr\") pod \"test-model-5d6f34-kserve-7d5896f6df-b9dtm\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") " pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:01.067655 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:01.067609    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:01.368291 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:01.368266    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"]
Sep 10 15:58:01.369970 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:58:01.369936    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14951fbb_f101_46ca_b3fc_5005137ae41b.slice/crio-6bb7eef5fef0316b4bebb1bc6662e3fb567f7c1bfa5f24c9c7c8962e75eaff87 WatchSource:0}: Error finding container 6bb7eef5fef0316b4bebb1bc6662e3fb567f7c1bfa5f24c9c7c8962e75eaff87: Status 404 returned error can't find the container with id 6bb7eef5fef0316b4bebb1bc6662e3fb567f7c1bfa5f24c9c7c8962e75eaff87
Sep 10 15:58:01.499655 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:01.499548    2572 patch_prober.go:28] interesting pod/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 15:58:01.499830 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:01.499649    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh" podUID="a56ab010-1a58-46ce-82c1-ab7232d408b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 15:58:01.868596 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:01.868552    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm" event={"ID":"14951fbb-f101-46ca-b3fc-5005137ae41b","Type":"ContainerStarted","Data":"be54a91c13170ddac5f0597d476704b632e3e5bf351c59c89c13b263d8107ad6"}
Sep 10 15:58:01.868596 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:01.868597    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm" event={"ID":"14951fbb-f101-46ca-b3fc-5005137ae41b","Type":"ContainerStarted","Data":"6bb7eef5fef0316b4bebb1bc6662e3fb567f7c1bfa5f24c9c7c8962e75eaff87"}
Sep 10 15:58:01.886048 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:01.885985    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm" podStartSLOduration=1.8859661189999999 podStartE2EDuration="1.885966119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:58:01.884170594 +0000 UTC m=+954.556580678" watchObservedRunningTime="2026-09-10 15:58:01.885966119 +0000 UTC m=+954.558376248"
Sep 10 15:58:02.500217 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:02.500159    2572 patch_prober.go:28] interesting pod/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 10 15:58:02.500446 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:02.500228    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh" podUID="a56ab010-1a58-46ce-82c1-ab7232d408b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 10 15:58:03.503693 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:03.503663    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:58:03.504140 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:03.503969    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:58:03.504730 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:03.504710    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:58:11.068253 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:11.068198    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:11.068711 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:11.068286    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:11.081108 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:11.081071    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:11.937399 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:11.937362    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:58:23.452610 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.452570    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"]
Sep 10 15:58:23.457691 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.457667    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:23.460761 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.460725    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-5c1d26\"/\"explicit-model-5c1d26-kserve-self-signed-certs\""
Sep 10 15:58:23.460761 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.460725    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-5c1d26\"/\"default-dockercfg-dqt5q\""
Sep 10 15:58:23.460761 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.460723    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-5c1d26\"/\"kube-root-ca.crt\""
Sep 10 15:58:23.461116 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.460803    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-5c1d26\"/\"openshift-service-ca.crt\""
Sep 10 15:58:23.464818 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.464790    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"]
Sep 10 15:58:23.621540 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.621493    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-model-cache\") pod \"explicit-model-5c1d26-kserve-556d7779ff-77nsw\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") " pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:23.621731 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.621630    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-dshm\") pod \"explicit-model-5c1d26-kserve-556d7779ff-77nsw\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") " pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:23.621731 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.621674    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40a19017-c6ff-494a-85bb-b18079774fd7-tls-certs\") pod \"explicit-model-5c1d26-kserve-556d7779ff-77nsw\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") " pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:23.621731 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.621700    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmngx\" (UniqueName: \"kubernetes.io/projected/40a19017-c6ff-494a-85bb-b18079774fd7-kube-api-access-mmngx\") pod \"explicit-model-5c1d26-kserve-556d7779ff-77nsw\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") " pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:23.621949 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.621813    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-tmp-dir\") pod \"explicit-model-5c1d26-kserve-556d7779ff-77nsw\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") " pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:23.621949 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.621901    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-home\") pod \"explicit-model-5c1d26-kserve-556d7779ff-77nsw\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") " pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:23.723120 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.723021    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40a19017-c6ff-494a-85bb-b18079774fd7-tls-certs\") pod \"explicit-model-5c1d26-kserve-556d7779ff-77nsw\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") " pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:23.723120 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.723065    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmngx\" (UniqueName: \"kubernetes.io/projected/40a19017-c6ff-494a-85bb-b18079774fd7-kube-api-access-mmngx\") pod \"explicit-model-5c1d26-kserve-556d7779ff-77nsw\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") " pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:23.723120 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.723110    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-tmp-dir\") pod \"explicit-model-5c1d26-kserve-556d7779ff-77nsw\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") " pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:23.723469 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.723180    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-home\") pod \"explicit-model-5c1d26-kserve-556d7779ff-77nsw\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") " pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:23.723469 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.723258    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-model-cache\") pod \"explicit-model-5c1d26-kserve-556d7779ff-77nsw\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") " pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:23.723469 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.723336    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-dshm\") pod \"explicit-model-5c1d26-kserve-556d7779ff-77nsw\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") " pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:23.723697 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.723669    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-tmp-dir\") pod \"explicit-model-5c1d26-kserve-556d7779ff-77nsw\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") " pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:23.723759 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.723682    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-home\") pod \"explicit-model-5c1d26-kserve-556d7779ff-77nsw\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") " pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:23.723810 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.723785    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-model-cache\") pod \"explicit-model-5c1d26-kserve-556d7779ff-77nsw\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") " pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:23.726141 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.726109    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40a19017-c6ff-494a-85bb-b18079774fd7-tls-certs\") pod \"explicit-model-5c1d26-kserve-556d7779ff-77nsw\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") " pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:23.726331 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.726313    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-dshm\") pod \"explicit-model-5c1d26-kserve-556d7779ff-77nsw\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") " pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:23.732118 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.732081    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmngx\" (UniqueName: \"kubernetes.io/projected/40a19017-c6ff-494a-85bb-b18079774fd7-kube-api-access-mmngx\") pod \"explicit-model-5c1d26-kserve-556d7779ff-77nsw\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") " pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:23.769040 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:23.768999    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:24.045327 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:24.045263    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"]
Sep 10 15:58:24.046443 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:58:24.046414    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40a19017_c6ff_494a_85bb_b18079774fd7.slice/crio-2772b61df39f98d9cb61bde3a850bdb3720c4e317f96dc3065264fe4dd10ae8c WatchSource:0}: Error finding container 2772b61df39f98d9cb61bde3a850bdb3720c4e317f96dc3065264fe4dd10ae8c: Status 404 returned error can't find the container with id 2772b61df39f98d9cb61bde3a850bdb3720c4e317f96dc3065264fe4dd10ae8c
Sep 10 15:58:25.006642 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:25.006607    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw" event={"ID":"40a19017-c6ff-494a-85bb-b18079774fd7","Type":"ContainerStarted","Data":"dabee1ff53c7073a73328c13a4c96063b00fef03532b4edcd5356adc778718cc"}
Sep 10 15:58:25.007046 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:25.006650    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw" event={"ID":"40a19017-c6ff-494a-85bb-b18079774fd7","Type":"ContainerStarted","Data":"2772b61df39f98d9cb61bde3a850bdb3720c4e317f96dc3065264fe4dd10ae8c"}
Sep 10 15:58:25.028807 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:25.028745    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw" podStartSLOduration=2.028727972 podStartE2EDuration="2.028727972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:58:25.026249571 +0000 UTC m=+977.698659657" watchObservedRunningTime="2026-09-10 15:58:25.028727972 +0000 UTC m=+977.701138096"
Sep 10 15:58:33.769905 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:33.769808    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:33.769905 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:33.769869    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:33.782745 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:33.782716    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:34.068925 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:34.068890    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:36.526038 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:36.525998    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"]
Sep 10 15:58:51.546351 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:51.546294    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw" podUID="40a19017-c6ff-494a-85bb-b18079774fd7" containerName="main" containerID="cri-o://dabee1ff53c7073a73328c13a4c96063b00fef03532b4edcd5356adc778718cc" gracePeriod=45
Sep 10 15:58:51.812370 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:51.812336    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:51.900996 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:51.900962    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmngx\" (UniqueName: \"kubernetes.io/projected/40a19017-c6ff-494a-85bb-b18079774fd7-kube-api-access-mmngx\") pod \"40a19017-c6ff-494a-85bb-b18079774fd7\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") "
Sep 10 15:58:51.900996 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:51.901007    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40a19017-c6ff-494a-85bb-b18079774fd7-tls-certs\") pod \"40a19017-c6ff-494a-85bb-b18079774fd7\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") "
Sep 10 15:58:51.901223 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:51.901038    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-tmp-dir\") pod \"40a19017-c6ff-494a-85bb-b18079774fd7\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") "
Sep 10 15:58:51.901223 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:51.901083    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-home\") pod \"40a19017-c6ff-494a-85bb-b18079774fd7\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") "
Sep 10 15:58:51.901223 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:51.901118    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-dshm\") pod \"40a19017-c6ff-494a-85bb-b18079774fd7\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") "
Sep 10 15:58:51.901223 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:51.901187    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-model-cache\") pod \"40a19017-c6ff-494a-85bb-b18079774fd7\" (UID: \"40a19017-c6ff-494a-85bb-b18079774fd7\") "
Sep 10 15:58:51.901452 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:51.901365    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40a19017-c6ff-494a-85bb-b18079774fd7" (UID: "40a19017-c6ff-494a-85bb-b18079774fd7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:58:51.901452 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:51.901372    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-home" (OuterVolumeSpecName: "home") pod "40a19017-c6ff-494a-85bb-b18079774fd7" (UID: "40a19017-c6ff-494a-85bb-b18079774fd7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:58:51.901564 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:51.901483    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-tmp-dir\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:58:51.901564 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:51.901527    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-home\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:58:51.901667 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:51.901583    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-model-cache" (OuterVolumeSpecName: "model-cache") pod "40a19017-c6ff-494a-85bb-b18079774fd7" (UID: "40a19017-c6ff-494a-85bb-b18079774fd7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:58:51.903522 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:51.903493    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a19017-c6ff-494a-85bb-b18079774fd7-kube-api-access-mmngx" (OuterVolumeSpecName: "kube-api-access-mmngx") pod "40a19017-c6ff-494a-85bb-b18079774fd7" (UID: "40a19017-c6ff-494a-85bb-b18079774fd7"). InnerVolumeSpecName "kube-api-access-mmngx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:58:51.903621 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:51.903515    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40a19017-c6ff-494a-85bb-b18079774fd7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40a19017-c6ff-494a-85bb-b18079774fd7" (UID: "40a19017-c6ff-494a-85bb-b18079774fd7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:58:51.903621 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:51.903566    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-dshm" (OuterVolumeSpecName: "dshm") pod "40a19017-c6ff-494a-85bb-b18079774fd7" (UID: "40a19017-c6ff-494a-85bb-b18079774fd7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:58:52.002388 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:52.002356    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-model-cache\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:58:52.002388 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:52.002389    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmngx\" (UniqueName: \"kubernetes.io/projected/40a19017-c6ff-494a-85bb-b18079774fd7-kube-api-access-mmngx\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:58:52.002547 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:52.002399    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40a19017-c6ff-494a-85bb-b18079774fd7-tls-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:58:52.002547 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:52.002409    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40a19017-c6ff-494a-85bb-b18079774fd7-dshm\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:58:52.169853 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:52.169757    2572 generic.go:356] "Generic (PLEG): container finished" podID="40a19017-c6ff-494a-85bb-b18079774fd7" containerID="dabee1ff53c7073a73328c13a4c96063b00fef03532b4edcd5356adc778718cc" exitCode=0
Sep 10 15:58:52.169853 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:52.169820    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"
Sep 10 15:58:52.170053 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:52.169846    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw" event={"ID":"40a19017-c6ff-494a-85bb-b18079774fd7","Type":"ContainerDied","Data":"dabee1ff53c7073a73328c13a4c96063b00fef03532b4edcd5356adc778718cc"}
Sep 10 15:58:52.170053 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:52.169893    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw" event={"ID":"40a19017-c6ff-494a-85bb-b18079774fd7","Type":"ContainerDied","Data":"2772b61df39f98d9cb61bde3a850bdb3720c4e317f96dc3065264fe4dd10ae8c"}
Sep 10 15:58:52.170053 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:52.169912    2572 scope.go:117] "RemoveContainer" containerID="dabee1ff53c7073a73328c13a4c96063b00fef03532b4edcd5356adc778718cc"
Sep 10 15:58:52.178981 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:52.178957    2572 scope.go:117] "RemoveContainer" containerID="dabee1ff53c7073a73328c13a4c96063b00fef03532b4edcd5356adc778718cc"
Sep 10 15:58:52.179338 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:58:52.179290    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dabee1ff53c7073a73328c13a4c96063b00fef03532b4edcd5356adc778718cc\": container with ID starting with dabee1ff53c7073a73328c13a4c96063b00fef03532b4edcd5356adc778718cc not found: ID does not exist" containerID="dabee1ff53c7073a73328c13a4c96063b00fef03532b4edcd5356adc778718cc"
Sep 10 15:58:52.179434 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:52.179349    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dabee1ff53c7073a73328c13a4c96063b00fef03532b4edcd5356adc778718cc"} err="failed to get container status \"dabee1ff53c7073a73328c13a4c96063b00fef03532b4edcd5356adc778718cc\": rpc error: code = NotFound desc = could not find container \"dabee1ff53c7073a73328c13a4c96063b00fef03532b4edcd5356adc778718cc\": container with ID starting with dabee1ff53c7073a73328c13a4c96063b00fef03532b4edcd5356adc778718cc not found: ID does not exist"
Sep 10 15:58:52.196343 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:52.196290    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"]
Sep 10 15:58:52.202714 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:52.202678    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-5c1d26/explicit-model-5c1d26-kserve-556d7779ff-77nsw"]
Sep 10 15:58:53.992288 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:53.992248    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40a19017-c6ff-494a-85bb-b18079774fd7" path="/var/lib/kubelet/pods/40a19017-c6ff-494a-85bb-b18079774fd7/volumes"
Sep 10 15:58:55.115909 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:55.115872    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"]
Sep 10 15:58:55.116345 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:58:55.116095    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh" podUID="a56ab010-1a58-46ce-82c1-ab7232d408b7" containerName="istio-proxy" containerID="cri-o://3c6165fa36a5cfafd0b0a34cde031a57c29cd639e34f3d235f083b584129ed8f" gracePeriod=30
Sep 10 15:59:00.375835 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.375808    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:59:00.477190 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.477074    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-podinfo\") pod \"a56ab010-1a58-46ce-82c1-ab7232d408b7\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") "
Sep 10 15:59:00.477190 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.477128    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qclww\" (UniqueName: \"kubernetes.io/projected/a56ab010-1a58-46ce-82c1-ab7232d408b7-kube-api-access-qclww\") pod \"a56ab010-1a58-46ce-82c1-ab7232d408b7\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") "
Sep 10 15:59:00.477190 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.477177    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-data\") pod \"a56ab010-1a58-46ce-82c1-ab7232d408b7\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") "
Sep 10 15:59:00.477536 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.477240    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a56ab010-1a58-46ce-82c1-ab7232d408b7-istiod-ca-cert\") pod \"a56ab010-1a58-46ce-82c1-ab7232d408b7\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") "
Sep 10 15:59:00.477536 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.477272    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-token\") pod \"a56ab010-1a58-46ce-82c1-ab7232d408b7\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") "
Sep 10 15:59:00.477536 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.477288    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-credential-socket\") pod \"a56ab010-1a58-46ce-82c1-ab7232d408b7\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") "
Sep 10 15:59:00.477536 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.477328    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-workload-socket\") pod \"a56ab010-1a58-46ce-82c1-ab7232d408b7\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") "
Sep 10 15:59:00.477536 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.477367    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-envoy\") pod \"a56ab010-1a58-46ce-82c1-ab7232d408b7\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") "
Sep 10 15:59:00.477806 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.477600    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-data" (OuterVolumeSpecName: "istio-data") pod "a56ab010-1a58-46ce-82c1-ab7232d408b7" (UID: "a56ab010-1a58-46ce-82c1-ab7232d408b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:00.477806 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.477627    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a56ab010-1a58-46ce-82c1-ab7232d408b7" (UID: "a56ab010-1a58-46ce-82c1-ab7232d408b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:00.477806 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.477665    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-workload-certs\") pod \"a56ab010-1a58-46ce-82c1-ab7232d408b7\" (UID: \"a56ab010-1a58-46ce-82c1-ab7232d408b7\") "
Sep 10 15:59:00.477806 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.477708    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a56ab010-1a58-46ce-82c1-ab7232d408b7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a56ab010-1a58-46ce-82c1-ab7232d408b7" (UID: "a56ab010-1a58-46ce-82c1-ab7232d408b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:59:00.477806 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.477748    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a56ab010-1a58-46ce-82c1-ab7232d408b7" (UID: "a56ab010-1a58-46ce-82c1-ab7232d408b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:00.478067 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.477910    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a56ab010-1a58-46ce-82c1-ab7232d408b7" (UID: "a56ab010-1a58-46ce-82c1-ab7232d408b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:00.478128 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.478064    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a56ab010-1a58-46ce-82c1-ab7232d408b7-istiod-ca-cert\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:00.478183 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.478086    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-credential-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:00.478183 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.478145    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-workload-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:00.478183 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.478160    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-workload-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:00.478388 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.478184    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-data\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:00.479652 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.479604    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-token" (OuterVolumeSpecName: "istio-token") pod "a56ab010-1a58-46ce-82c1-ab7232d408b7" (UID: "a56ab010-1a58-46ce-82c1-ab7232d408b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:59:00.479783 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.479652    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a56ab010-1a58-46ce-82c1-ab7232d408b7" (UID: "a56ab010-1a58-46ce-82c1-ab7232d408b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:59:00.479783 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.479687    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a56ab010-1a58-46ce-82c1-ab7232d408b7" (UID: "a56ab010-1a58-46ce-82c1-ab7232d408b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:00.479783 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.479699    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a56ab010-1a58-46ce-82c1-ab7232d408b7-kube-api-access-qclww" (OuterVolumeSpecName: "kube-api-access-qclww") pod "a56ab010-1a58-46ce-82c1-ab7232d408b7" (UID: "a56ab010-1a58-46ce-82c1-ab7232d408b7"). InnerVolumeSpecName "kube-api-access-qclww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:59:00.578732 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.578692    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-token\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:00.578732 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.578725    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-envoy\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:00.578732 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.578735    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a56ab010-1a58-46ce-82c1-ab7232d408b7-istio-podinfo\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:00.578990 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:00.578744    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qclww\" (UniqueName: \"kubernetes.io/projected/a56ab010-1a58-46ce-82c1-ab7232d408b7-kube-api-access-qclww\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:01.234430 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:01.234393    2572 generic.go:356] "Generic (PLEG): container finished" podID="a56ab010-1a58-46ce-82c1-ab7232d408b7" containerID="3c6165fa36a5cfafd0b0a34cde031a57c29cd639e34f3d235f083b584129ed8f" exitCode=0
Sep 10 15:59:01.234655 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:01.234443    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"
Sep 10 15:59:01.234655 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:01.234478    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh" event={"ID":"a56ab010-1a58-46ce-82c1-ab7232d408b7","Type":"ContainerDied","Data":"3c6165fa36a5cfafd0b0a34cde031a57c29cd639e34f3d235f083b584129ed8f"}
Sep 10 15:59:01.234655 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:01.234546    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh" event={"ID":"a56ab010-1a58-46ce-82c1-ab7232d408b7","Type":"ContainerDied","Data":"bd0845fc430287baf407be1bd5a3e9af801fb970392b0a7ea539548ac616f15a"}
Sep 10 15:59:01.234655 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:01.234576    2572 scope.go:117] "RemoveContainer" containerID="3c6165fa36a5cfafd0b0a34cde031a57c29cd639e34f3d235f083b584129ed8f"
Sep 10 15:59:01.245078 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:01.245056    2572 scope.go:117] "RemoveContainer" containerID="202c32ea149c1a093ad775236e5023a8a7658835b46d811d28bdd0a3e4fee087"
Sep 10 15:59:01.254143 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:01.254125    2572 scope.go:117] "RemoveContainer" containerID="3c6165fa36a5cfafd0b0a34cde031a57c29cd639e34f3d235f083b584129ed8f"
Sep 10 15:59:01.254436 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:59:01.254417    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c6165fa36a5cfafd0b0a34cde031a57c29cd639e34f3d235f083b584129ed8f\": container with ID starting with 3c6165fa36a5cfafd0b0a34cde031a57c29cd639e34f3d235f083b584129ed8f not found: ID does not exist" containerID="3c6165fa36a5cfafd0b0a34cde031a57c29cd639e34f3d235f083b584129ed8f"
Sep 10 15:59:01.254487 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:01.254446    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c6165fa36a5cfafd0b0a34cde031a57c29cd639e34f3d235f083b584129ed8f"} err="failed to get container status \"3c6165fa36a5cfafd0b0a34cde031a57c29cd639e34f3d235f083b584129ed8f\": rpc error: code = NotFound desc = could not find container \"3c6165fa36a5cfafd0b0a34cde031a57c29cd639e34f3d235f083b584129ed8f\": container with ID starting with 3c6165fa36a5cfafd0b0a34cde031a57c29cd639e34f3d235f083b584129ed8f not found: ID does not exist"
Sep 10 15:59:01.254487 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:01.254463    2572 scope.go:117] "RemoveContainer" containerID="202c32ea149c1a093ad775236e5023a8a7658835b46d811d28bdd0a3e4fee087"
Sep 10 15:59:01.254694 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:59:01.254677    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"202c32ea149c1a093ad775236e5023a8a7658835b46d811d28bdd0a3e4fee087\": container with ID starting with 202c32ea149c1a093ad775236e5023a8a7658835b46d811d28bdd0a3e4fee087 not found: ID does not exist" containerID="202c32ea149c1a093ad775236e5023a8a7658835b46d811d28bdd0a3e4fee087"
Sep 10 15:59:01.254731 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:01.254699    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"202c32ea149c1a093ad775236e5023a8a7658835b46d811d28bdd0a3e4fee087"} err="failed to get container status \"202c32ea149c1a093ad775236e5023a8a7658835b46d811d28bdd0a3e4fee087\": rpc error: code = NotFound desc = could not find container \"202c32ea149c1a093ad775236e5023a8a7658835b46d811d28bdd0a3e4fee087\": container with ID starting with 202c32ea149c1a093ad775236e5023a8a7658835b46d811d28bdd0a3e4fee087 not found: ID does not exist"
Sep 10 15:59:01.260908 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:01.260880    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"]
Sep 10 15:59:01.265987 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:01.265962    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-5c1d26-openshift-default-58d5d5886d-8qxjh"]
Sep 10 15:59:01.993608 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:01.993572    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a56ab010-1a58-46ce-82c1-ab7232d408b7" path="/var/lib/kubelet/pods/a56ab010-1a58-46ce-82c1-ab7232d408b7/volumes"
Sep 10 15:59:05.013810 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.013777    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"]
Sep 10 15:59:05.014259 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.014147    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a56ab010-1a58-46ce-82c1-ab7232d408b7" containerName="istio-proxy"
Sep 10 15:59:05.014259 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.014162    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a56ab010-1a58-46ce-82c1-ab7232d408b7" containerName="istio-proxy"
Sep 10 15:59:05.014259 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.014177    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40a19017-c6ff-494a-85bb-b18079774fd7" containerName="main"
Sep 10 15:59:05.014259 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.014182    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="40a19017-c6ff-494a-85bb-b18079774fd7" containerName="main"
Sep 10 15:59:05.014259 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.014244    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a56ab010-1a58-46ce-82c1-ab7232d408b7" containerName="istio-proxy"
Sep 10 15:59:05.014259 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.014255    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a56ab010-1a58-46ce-82c1-ab7232d408b7" containerName="istio-proxy"
Sep 10 15:59:05.014259 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.014263    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="40a19017-c6ff-494a-85bb-b18079774fd7" containerName="main"
Sep 10 15:59:05.018799 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.018775    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.023203 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.023176    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-066ed8\"/\"noten-model-066ed8-kserve-self-signed-certs\""
Sep 10 15:59:05.023595 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.023176    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-066ed8\"/\"kube-root-ca.crt\""
Sep 10 15:59:05.023595 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.023238    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-066ed8\"/\"default-dockercfg-hn2kg\""
Sep 10 15:59:05.046260 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.046228    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-066ed8\"/\"openshift-service-ca.crt\""
Sep 10 15:59:05.050596 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.050560    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"]
Sep 10 15:59:05.119385 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.119343    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-model-cache\") pod \"noten-model-066ed8-kserve-6968887d99-4bt6k\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") " pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.119601 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.119410    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-home\") pod \"noten-model-066ed8-kserve-6968887d99-4bt6k\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") " pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.119601 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.119431    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-dshm\") pod \"noten-model-066ed8-kserve-6968887d99-4bt6k\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") " pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.119601 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.119449    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697591ba-19c3-4a70-80d7-f73cc6985292-tls-certs\") pod \"noten-model-066ed8-kserve-6968887d99-4bt6k\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") " pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.119601 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.119465    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2pcq\" (UniqueName: \"kubernetes.io/projected/697591ba-19c3-4a70-80d7-f73cc6985292-kube-api-access-c2pcq\") pod \"noten-model-066ed8-kserve-6968887d99-4bt6k\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") " pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.119601 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.119538    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-tmp-dir\") pod \"noten-model-066ed8-kserve-6968887d99-4bt6k\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") " pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.220841 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.220797    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-model-cache\") pod \"noten-model-066ed8-kserve-6968887d99-4bt6k\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") " pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.221045 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.220920    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-home\") pod \"noten-model-066ed8-kserve-6968887d99-4bt6k\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") " pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.221045 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.220952    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-dshm\") pod \"noten-model-066ed8-kserve-6968887d99-4bt6k\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") " pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.221045 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.220976    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697591ba-19c3-4a70-80d7-f73cc6985292-tls-certs\") pod \"noten-model-066ed8-kserve-6968887d99-4bt6k\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") " pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.221045 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.221001    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2pcq\" (UniqueName: \"kubernetes.io/projected/697591ba-19c3-4a70-80d7-f73cc6985292-kube-api-access-c2pcq\") pod \"noten-model-066ed8-kserve-6968887d99-4bt6k\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") " pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.221277 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.221051    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-tmp-dir\") pod \"noten-model-066ed8-kserve-6968887d99-4bt6k\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") " pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.221432 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.221296    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-model-cache\") pod \"noten-model-066ed8-kserve-6968887d99-4bt6k\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") " pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.221578 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.221439    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-home\") pod \"noten-model-066ed8-kserve-6968887d99-4bt6k\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") " pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.221966 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.221936    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-tmp-dir\") pod \"noten-model-066ed8-kserve-6968887d99-4bt6k\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") " pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.223825 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.223803    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-dshm\") pod \"noten-model-066ed8-kserve-6968887d99-4bt6k\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") " pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.224402 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.224381    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697591ba-19c3-4a70-80d7-f73cc6985292-tls-certs\") pod \"noten-model-066ed8-kserve-6968887d99-4bt6k\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") " pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.231461 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.231436    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2pcq\" (UniqueName: \"kubernetes.io/projected/697591ba-19c3-4a70-80d7-f73cc6985292-kube-api-access-c2pcq\") pod \"noten-model-066ed8-kserve-6968887d99-4bt6k\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") " pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.328163 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.328123    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:05.602826 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:05.602796    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"]
Sep 10 15:59:05.604721 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:59:05.604692    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod697591ba_19c3_4a70_80d7_f73cc6985292.slice/crio-ceeb201c669a59c44d322decd4d651288e0f48a1cddf0e5320e00767eae6c3c5 WatchSource:0}: Error finding container ceeb201c669a59c44d322decd4d651288e0f48a1cddf0e5320e00767eae6c3c5: Status 404 returned error can't find the container with id ceeb201c669a59c44d322decd4d651288e0f48a1cddf0e5320e00767eae6c3c5
Sep 10 15:59:06.265949 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:06.265913    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k" event={"ID":"697591ba-19c3-4a70-80d7-f73cc6985292","Type":"ContainerStarted","Data":"9cf372ab8b468c9d64904d527f573ffac8274c3264f91452fe545e7ea5b4b46d"}
Sep 10 15:59:06.266419 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:06.265960    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k" event={"ID":"697591ba-19c3-4a70-80d7-f73cc6985292","Type":"ContainerStarted","Data":"ceeb201c669a59c44d322decd4d651288e0f48a1cddf0e5320e00767eae6c3c5"}
Sep 10 15:59:06.291201 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:06.291148    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k" podStartSLOduration=2.291130182 podStartE2EDuration="2.291130182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:59:06.288666923 +0000 UTC m=+1018.961077007" watchObservedRunningTime="2026-09-10 15:59:06.291130182 +0000 UTC m=+1018.963540264"
Sep 10 15:59:15.329155 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:15.329112    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:15.329644 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:15.329173    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:15.344240 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:15.344213    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:16.334393 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:16.334356    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:17.066483 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:17.066450    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"]
Sep 10 15:59:17.908857 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:17.908815    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"]
Sep 10 15:59:19.074539 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:19.074506    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"]
Sep 10 15:59:19.195600 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:19.195560    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj"]
Sep 10 15:59:19.195952 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:19.195940    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a56ab010-1a58-46ce-82c1-ab7232d408b7" containerName="istio-proxy"
Sep 10 15:59:19.195999 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:19.195955    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a56ab010-1a58-46ce-82c1-ab7232d408b7" containerName="istio-proxy"
Sep 10 15:59:19.200465 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:19.200439    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj"
Sep 10 15:59:19.203475 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:19.203451    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 15:59:19.203613 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:19.203474    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\""
Sep 10 15:59:19.211081 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:19.211055    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj"]
Sep 10 15:59:19.255322 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:19.255259    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/075c5939-6a71-4cc6-87b7-0aa79928b071-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj\" (UID: \"075c5939-6a71-4cc6-87b7-0aa79928b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj"
Sep 10 15:59:19.356845 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:19.356752    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/075c5939-6a71-4cc6-87b7-0aa79928b071-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj\" (UID: \"075c5939-6a71-4cc6-87b7-0aa79928b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj"
Sep 10 15:59:19.357407 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:19.357385    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/075c5939-6a71-4cc6-87b7-0aa79928b071-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj\" (UID: \"075c5939-6a71-4cc6-87b7-0aa79928b071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj"
Sep 10 15:59:19.512359 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:19.512329    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj"
Sep 10 15:59:19.794058 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:19.794033    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj"]
Sep 10 15:59:19.795755 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:59:19.795727    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075c5939_6a71_4cc6_87b7_0aa79928b071.slice/crio-1197be8463ca11fff7ffdaba96ccb38d6b7302daaaca8e6f5cc6ae9da5bf0817 WatchSource:0}: Error finding container 1197be8463ca11fff7ffdaba96ccb38d6b7302daaaca8e6f5cc6ae9da5bf0817: Status 404 returned error can't find the container with id 1197be8463ca11fff7ffdaba96ccb38d6b7302daaaca8e6f5cc6ae9da5bf0817
Sep 10 15:59:20.345279 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:20.345244    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj" event={"ID":"075c5939-6a71-4cc6-87b7-0aa79928b071","Type":"ContainerStarted","Data":"b389090ca99f30161f19abc5f361b6b17d7bda3dea5617cbe30fb15808f2c6a9"}
Sep 10 15:59:20.345707 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:20.345316    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj" event={"ID":"075c5939-6a71-4cc6-87b7-0aa79928b071","Type":"ContainerStarted","Data":"1197be8463ca11fff7ffdaba96ccb38d6b7302daaaca8e6f5cc6ae9da5bf0817"}
Sep 10 15:59:20.361479 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:20.361426    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj" podStartSLOduration=1.361410792 podStartE2EDuration="1.361410792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:59:20.360669136 +0000 UTC m=+1033.033079223" watchObservedRunningTime="2026-09-10 15:59:20.361410792 +0000 UTC m=+1033.033820890"
Sep 10 15:59:21.352451 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:21.352414    2572 generic.go:356] "Generic (PLEG): container finished" podID="075c5939-6a71-4cc6-87b7-0aa79928b071" containerID="b389090ca99f30161f19abc5f361b6b17d7bda3dea5617cbe30fb15808f2c6a9" exitCode=0
Sep 10 15:59:21.352915 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:21.352506    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj" event={"ID":"075c5939-6a71-4cc6-87b7-0aa79928b071","Type":"ContainerDied","Data":"b389090ca99f30161f19abc5f361b6b17d7bda3dea5617cbe30fb15808f2c6a9"}
Sep 10 15:59:22.487037 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:22.487009    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj"
Sep 10 15:59:22.588171 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:22.588131    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/075c5939-6a71-4cc6-87b7-0aa79928b071-maas-api-service-ca\") pod \"075c5939-6a71-4cc6-87b7-0aa79928b071\" (UID: \"075c5939-6a71-4cc6-87b7-0aa79928b071\") "
Sep 10 15:59:22.588537 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:22.588509    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/075c5939-6a71-4cc6-87b7-0aa79928b071-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "075c5939-6a71-4cc6-87b7-0aa79928b071" (UID: "075c5939-6a71-4cc6-87b7-0aa79928b071"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:59:22.689032 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:22.688941    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/075c5939-6a71-4cc6-87b7-0aa79928b071-maas-api-service-ca\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:23.366979 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:23.366935    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj" event={"ID":"075c5939-6a71-4cc6-87b7-0aa79928b071","Type":"ContainerDied","Data":"1197be8463ca11fff7ffdaba96ccb38d6b7302daaaca8e6f5cc6ae9da5bf0817"}
Sep 10 15:59:23.367273 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:23.367253    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1197be8463ca11fff7ffdaba96ccb38d6b7302daaaca8e6f5cc6ae9da5bf0817"
Sep 10 15:59:23.367475 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:23.367461    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj"
Sep 10 15:59:30.058770 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:30.058733    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj"]
Sep 10 15:59:30.066020 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:30.065991    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-789be7-qdlfj"]
Sep 10 15:59:31.992132 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:31.992092    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="075c5939-6a71-4cc6-87b7-0aa79928b071" path="/var/lib/kubelet/pods/075c5939-6a71-4cc6-87b7-0aa79928b071/volumes"
Sep 10 15:59:32.086000 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.085955    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5" podUID="4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a" containerName="main" containerID="cri-o://47717528425b2250763b4bff9aeaf9d25b1399ad5d545bef93b0d373cd53bc0e" gracePeriod=45
Sep 10 15:59:32.333517 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.333483    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:59:32.417437 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.417390    2572 generic.go:356] "Generic (PLEG): container finished" podID="4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a" containerID="47717528425b2250763b4bff9aeaf9d25b1399ad5d545bef93b0d373cd53bc0e" exitCode=0
Sep 10 15:59:32.417437 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.417430    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"
Sep 10 15:59:32.417659 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.417450    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5" event={"ID":"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a","Type":"ContainerDied","Data":"47717528425b2250763b4bff9aeaf9d25b1399ad5d545bef93b0d373cd53bc0e"}
Sep 10 15:59:32.417659 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.417508    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5" event={"ID":"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a","Type":"ContainerDied","Data":"474cd2e59cee928c9b30960b7651d469e819b68148f67e18f466b64844f7f224"}
Sep 10 15:59:32.417659 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.417533    2572 scope.go:117] "RemoveContainer" containerID="47717528425b2250763b4bff9aeaf9d25b1399ad5d545bef93b0d373cd53bc0e"
Sep 10 15:59:32.427320 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.427282    2572 scope.go:117] "RemoveContainer" containerID="47717528425b2250763b4bff9aeaf9d25b1399ad5d545bef93b0d373cd53bc0e"
Sep 10 15:59:32.427701 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:59:32.427679    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47717528425b2250763b4bff9aeaf9d25b1399ad5d545bef93b0d373cd53bc0e\": container with ID starting with 47717528425b2250763b4bff9aeaf9d25b1399ad5d545bef93b0d373cd53bc0e not found: ID does not exist" containerID="47717528425b2250763b4bff9aeaf9d25b1399ad5d545bef93b0d373cd53bc0e"
Sep 10 15:59:32.427757 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.427713    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47717528425b2250763b4bff9aeaf9d25b1399ad5d545bef93b0d373cd53bc0e"} err="failed to get container status \"47717528425b2250763b4bff9aeaf9d25b1399ad5d545bef93b0d373cd53bc0e\": rpc error: code = NotFound desc = could not find container \"47717528425b2250763b4bff9aeaf9d25b1399ad5d545bef93b0d373cd53bc0e\": container with ID starting with 47717528425b2250763b4bff9aeaf9d25b1399ad5d545bef93b0d373cd53bc0e not found: ID does not exist"
Sep 10 15:59:32.485109 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.485073    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-tmp-dir\") pod \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") "
Sep 10 15:59:32.485279 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.485121    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-dshm\") pod \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") "
Sep 10 15:59:32.485279 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.485146    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-tls-certs\") pod \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") "
Sep 10 15:59:32.485279 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.485218    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k25pd\" (UniqueName: \"kubernetes.io/projected/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-kube-api-access-k25pd\") pod \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") "
Sep 10 15:59:32.485279 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.485269    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-home\") pod \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") "
Sep 10 15:59:32.485538 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.485284    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-model-cache\") pod \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\" (UID: \"4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a\") "
Sep 10 15:59:32.485538 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.485462    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a" (UID: "4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:32.485640 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.485570    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-home" (OuterVolumeSpecName: "home") pod "4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a" (UID: "4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:32.485693 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.485638    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-model-cache" (OuterVolumeSpecName: "model-cache") pod "4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a" (UID: "4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:32.485693 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.485662    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-tmp-dir\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:32.485779 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.485704    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-home\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:32.487588 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.487560    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-dshm" (OuterVolumeSpecName: "dshm") pod "4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a" (UID: "4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:32.487588 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.487572    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a" (UID: "4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:59:32.487738 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.487595    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-kube-api-access-k25pd" (OuterVolumeSpecName: "kube-api-access-k25pd") pod "4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a" (UID: "4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a"). InnerVolumeSpecName "kube-api-access-k25pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:59:32.587011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.586971    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-dshm\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:32.587011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.587005    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-tls-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:32.587011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.587015    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k25pd\" (UniqueName: \"kubernetes.io/projected/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-kube-api-access-k25pd\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:32.587267 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.587026    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a-model-cache\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:32.746661 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.746627    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"]
Sep 10 15:59:32.758079 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:32.758040    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-789be7/test-model-789be7-kserve-77574677d9-pf7g5"]
Sep 10 15:59:33.352527 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.352467    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k" podUID="697591ba-19c3-4a70-80d7-f73cc6985292" containerName="main" containerID="cri-o://9cf372ab8b468c9d64904d527f573ffac8274c3264f91452fe545e7ea5b4b46d" gracePeriod=45
Sep 10 15:59:33.600923 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.600894    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:33.695956 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.695858    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-dshm\") pod \"697591ba-19c3-4a70-80d7-f73cc6985292\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") "
Sep 10 15:59:33.695956 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.695907    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2pcq\" (UniqueName: \"kubernetes.io/projected/697591ba-19c3-4a70-80d7-f73cc6985292-kube-api-access-c2pcq\") pod \"697591ba-19c3-4a70-80d7-f73cc6985292\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") "
Sep 10 15:59:33.695956 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.695931    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-tmp-dir\") pod \"697591ba-19c3-4a70-80d7-f73cc6985292\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") "
Sep 10 15:59:33.696252 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.696020    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697591ba-19c3-4a70-80d7-f73cc6985292-tls-certs\") pod \"697591ba-19c3-4a70-80d7-f73cc6985292\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") "
Sep 10 15:59:33.696252 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.696066    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-model-cache\") pod \"697591ba-19c3-4a70-80d7-f73cc6985292\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") "
Sep 10 15:59:33.696252 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.696103    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-home\") pod \"697591ba-19c3-4a70-80d7-f73cc6985292\" (UID: \"697591ba-19c3-4a70-80d7-f73cc6985292\") "
Sep 10 15:59:33.696434 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.696258    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "697591ba-19c3-4a70-80d7-f73cc6985292" (UID: "697591ba-19c3-4a70-80d7-f73cc6985292"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:33.696434 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.696374    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-model-cache" (OuterVolumeSpecName: "model-cache") pod "697591ba-19c3-4a70-80d7-f73cc6985292" (UID: "697591ba-19c3-4a70-80d7-f73cc6985292"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:33.696515 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.696460    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-home" (OuterVolumeSpecName: "home") pod "697591ba-19c3-4a70-80d7-f73cc6985292" (UID: "697591ba-19c3-4a70-80d7-f73cc6985292"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:33.696554 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.696531    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-tmp-dir\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:33.696554 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.696546    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-model-cache\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:33.698274 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.698240    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/697591ba-19c3-4a70-80d7-f73cc6985292-kube-api-access-c2pcq" (OuterVolumeSpecName: "kube-api-access-c2pcq") pod "697591ba-19c3-4a70-80d7-f73cc6985292" (UID: "697591ba-19c3-4a70-80d7-f73cc6985292"). InnerVolumeSpecName "kube-api-access-c2pcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:59:33.698573 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.698546    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/697591ba-19c3-4a70-80d7-f73cc6985292-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "697591ba-19c3-4a70-80d7-f73cc6985292" (UID: "697591ba-19c3-4a70-80d7-f73cc6985292"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:59:33.698666 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.698546    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-dshm" (OuterVolumeSpecName: "dshm") pod "697591ba-19c3-4a70-80d7-f73cc6985292" (UID: "697591ba-19c3-4a70-80d7-f73cc6985292"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:33.797168 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.797125    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697591ba-19c3-4a70-80d7-f73cc6985292-tls-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:33.797168 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.797165    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-home\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:33.797168 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.797175    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697591ba-19c3-4a70-80d7-f73cc6985292-dshm\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:33.797471 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.797187    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2pcq\" (UniqueName: \"kubernetes.io/projected/697591ba-19c3-4a70-80d7-f73cc6985292-kube-api-access-c2pcq\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:33.998813 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:33.998725    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a" path="/var/lib/kubelet/pods/4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a/volumes"
Sep 10 15:59:34.094491 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.094445    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm" podUID="14951fbb-f101-46ca-b3fc-5005137ae41b" containerName="main" containerID="cri-o://be54a91c13170ddac5f0597d476704b632e3e5bf351c59c89c13b263d8107ad6" gracePeriod=45
Sep 10 15:59:34.336198 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.336170    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:59:34.432143 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.432106    2572 generic.go:356] "Generic (PLEG): container finished" podID="14951fbb-f101-46ca-b3fc-5005137ae41b" containerID="be54a91c13170ddac5f0597d476704b632e3e5bf351c59c89c13b263d8107ad6" exitCode=0
Sep 10 15:59:34.432615 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.432150    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"
Sep 10 15:59:34.432615 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.432192    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm" event={"ID":"14951fbb-f101-46ca-b3fc-5005137ae41b","Type":"ContainerDied","Data":"be54a91c13170ddac5f0597d476704b632e3e5bf351c59c89c13b263d8107ad6"}
Sep 10 15:59:34.432615 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.432239    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm" event={"ID":"14951fbb-f101-46ca-b3fc-5005137ae41b","Type":"ContainerDied","Data":"6bb7eef5fef0316b4bebb1bc6662e3fb567f7c1bfa5f24c9c7c8962e75eaff87"}
Sep 10 15:59:34.432615 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.432259    2572 scope.go:117] "RemoveContainer" containerID="be54a91c13170ddac5f0597d476704b632e3e5bf351c59c89c13b263d8107ad6"
Sep 10 15:59:34.433665 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.433642    2572 generic.go:356] "Generic (PLEG): container finished" podID="697591ba-19c3-4a70-80d7-f73cc6985292" containerID="9cf372ab8b468c9d64904d527f573ffac8274c3264f91452fe545e7ea5b4b46d" exitCode=0
Sep 10 15:59:34.433724 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.433699    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"
Sep 10 15:59:34.433724 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.433702    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k" event={"ID":"697591ba-19c3-4a70-80d7-f73cc6985292","Type":"ContainerDied","Data":"9cf372ab8b468c9d64904d527f573ffac8274c3264f91452fe545e7ea5b4b46d"}
Sep 10 15:59:34.433807 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.433734    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k" event={"ID":"697591ba-19c3-4a70-80d7-f73cc6985292","Type":"ContainerDied","Data":"ceeb201c669a59c44d322decd4d651288e0f48a1cddf0e5320e00767eae6c3c5"}
Sep 10 15:59:34.441378 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.441358    2572 scope.go:117] "RemoveContainer" containerID="be54a91c13170ddac5f0597d476704b632e3e5bf351c59c89c13b263d8107ad6"
Sep 10 15:59:34.441700 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:59:34.441681    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be54a91c13170ddac5f0597d476704b632e3e5bf351c59c89c13b263d8107ad6\": container with ID starting with be54a91c13170ddac5f0597d476704b632e3e5bf351c59c89c13b263d8107ad6 not found: ID does not exist" containerID="be54a91c13170ddac5f0597d476704b632e3e5bf351c59c89c13b263d8107ad6"
Sep 10 15:59:34.441760 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.441711    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be54a91c13170ddac5f0597d476704b632e3e5bf351c59c89c13b263d8107ad6"} err="failed to get container status \"be54a91c13170ddac5f0597d476704b632e3e5bf351c59c89c13b263d8107ad6\": rpc error: code = NotFound desc = could not find container \"be54a91c13170ddac5f0597d476704b632e3e5bf351c59c89c13b263d8107ad6\": container with ID starting with be54a91c13170ddac5f0597d476704b632e3e5bf351c59c89c13b263d8107ad6 not found: ID does not exist"
Sep 10 15:59:34.441760 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.441730    2572 scope.go:117] "RemoveContainer" containerID="9cf372ab8b468c9d64904d527f573ffac8274c3264f91452fe545e7ea5b4b46d"
Sep 10 15:59:34.450213 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.450187    2572 scope.go:117] "RemoveContainer" containerID="9cf372ab8b468c9d64904d527f573ffac8274c3264f91452fe545e7ea5b4b46d"
Sep 10 15:59:34.450581 ip-10-0-134-36 kubenswrapper[2572]: E0910 15:59:34.450559    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cf372ab8b468c9d64904d527f573ffac8274c3264f91452fe545e7ea5b4b46d\": container with ID starting with 9cf372ab8b468c9d64904d527f573ffac8274c3264f91452fe545e7ea5b4b46d not found: ID does not exist" containerID="9cf372ab8b468c9d64904d527f573ffac8274c3264f91452fe545e7ea5b4b46d"
Sep 10 15:59:34.450656 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.450592    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cf372ab8b468c9d64904d527f573ffac8274c3264f91452fe545e7ea5b4b46d"} err="failed to get container status \"9cf372ab8b468c9d64904d527f573ffac8274c3264f91452fe545e7ea5b4b46d\": rpc error: code = NotFound desc = could not find container \"9cf372ab8b468c9d64904d527f573ffac8274c3264f91452fe545e7ea5b4b46d\": container with ID starting with 9cf372ab8b468c9d64904d527f573ffac8274c3264f91452fe545e7ea5b4b46d not found: ID does not exist"
Sep 10 15:59:34.457231 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.457202    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"]
Sep 10 15:59:34.461797 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.461769    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-066ed8/noten-model-066ed8-kserve-6968887d99-4bt6k"]
Sep 10 15:59:34.501547 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.501514    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-model-cache\") pod \"14951fbb-f101-46ca-b3fc-5005137ae41b\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") "
Sep 10 15:59:34.501697 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.501606    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14951fbb-f101-46ca-b3fc-5005137ae41b-tls-certs\") pod \"14951fbb-f101-46ca-b3fc-5005137ae41b\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") "
Sep 10 15:59:34.501697 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.501624    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-home\") pod \"14951fbb-f101-46ca-b3fc-5005137ae41b\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") "
Sep 10 15:59:34.501697 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.501653    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d94hr\" (UniqueName: \"kubernetes.io/projected/14951fbb-f101-46ca-b3fc-5005137ae41b-kube-api-access-d94hr\") pod \"14951fbb-f101-46ca-b3fc-5005137ae41b\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") "
Sep 10 15:59:34.501697 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.501682    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-dshm\") pod \"14951fbb-f101-46ca-b3fc-5005137ae41b\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") "
Sep 10 15:59:34.501898 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.501714    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-tmp-dir\") pod \"14951fbb-f101-46ca-b3fc-5005137ae41b\" (UID: \"14951fbb-f101-46ca-b3fc-5005137ae41b\") "
Sep 10 15:59:34.501898 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.501839    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-model-cache" (OuterVolumeSpecName: "model-cache") pod "14951fbb-f101-46ca-b3fc-5005137ae41b" (UID: "14951fbb-f101-46ca-b3fc-5005137ae41b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:34.502011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.501938    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-home" (OuterVolumeSpecName: "home") pod "14951fbb-f101-46ca-b3fc-5005137ae41b" (UID: "14951fbb-f101-46ca-b3fc-5005137ae41b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:34.502011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.501977    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14951fbb-f101-46ca-b3fc-5005137ae41b" (UID: "14951fbb-f101-46ca-b3fc-5005137ae41b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:34.502011 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.501994    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-model-cache\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:34.504102 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.504071    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14951fbb-f101-46ca-b3fc-5005137ae41b-kube-api-access-d94hr" (OuterVolumeSpecName: "kube-api-access-d94hr") pod "14951fbb-f101-46ca-b3fc-5005137ae41b" (UID: "14951fbb-f101-46ca-b3fc-5005137ae41b"). InnerVolumeSpecName "kube-api-access-d94hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:59:34.504102 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.504092    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-dshm" (OuterVolumeSpecName: "dshm") pod "14951fbb-f101-46ca-b3fc-5005137ae41b" (UID: "14951fbb-f101-46ca-b3fc-5005137ae41b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:34.504254 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.504071    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14951fbb-f101-46ca-b3fc-5005137ae41b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14951fbb-f101-46ca-b3fc-5005137ae41b" (UID: "14951fbb-f101-46ca-b3fc-5005137ae41b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:59:34.602790 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.602738    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14951fbb-f101-46ca-b3fc-5005137ae41b-tls-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:34.602790 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.602779    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-home\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:34.602790 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.602789    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d94hr\" (UniqueName: \"kubernetes.io/projected/14951fbb-f101-46ca-b3fc-5005137ae41b-kube-api-access-d94hr\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:34.602790 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.602800    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-dshm\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:34.603077 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.602832    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14951fbb-f101-46ca-b3fc-5005137ae41b-tmp-dir\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:34.755435 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.755385    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"]
Sep 10 15:59:34.761190 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:34.761155    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5d6f34/test-model-5d6f34-kserve-7d5896f6df-b9dtm"]
Sep 10 15:59:35.992191 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:35.992153    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14951fbb-f101-46ca-b3fc-5005137ae41b" path="/var/lib/kubelet/pods/14951fbb-f101-46ca-b3fc-5005137ae41b/volumes"
Sep 10 15:59:35.992697 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:35.992582    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="697591ba-19c3-4a70-80d7-f73cc6985292" path="/var/lib/kubelet/pods/697591ba-19c3-4a70-80d7-f73cc6985292/volumes"
Sep 10 15:59:37.843063 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:37.843023    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d6f34-gbgnz"]
Sep 10 15:59:37.843568 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:37.843379    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="697591ba-19c3-4a70-80d7-f73cc6985292" containerName="main"
Sep 10 15:59:37.843568 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:37.843392    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="697591ba-19c3-4a70-80d7-f73cc6985292" containerName="main"
Sep 10 15:59:37.843568 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:37.843408    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="075c5939-6a71-4cc6-87b7-0aa79928b071" containerName="revoke-keys"
Sep 10 15:59:37.843568 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:37.843414    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="075c5939-6a71-4cc6-87b7-0aa79928b071" containerName="revoke-keys"
Sep 10 15:59:37.843568 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:37.843431    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a" containerName="main"
Sep 10 15:59:37.843568 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:37.843437    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a" containerName="main"
Sep 10 15:59:37.843568 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:37.843444    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14951fbb-f101-46ca-b3fc-5005137ae41b" containerName="main"
Sep 10 15:59:37.843568 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:37.843449    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="14951fbb-f101-46ca-b3fc-5005137ae41b" containerName="main"
Sep 10 15:59:37.843568 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:37.843501    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fd169cb-2e7f-4bfb-b542-3b7e086b7b9a" containerName="main"
Sep 10 15:59:37.843568 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:37.843509    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="697591ba-19c3-4a70-80d7-f73cc6985292" containerName="main"
Sep 10 15:59:37.843568 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:37.843515    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="14951fbb-f101-46ca-b3fc-5005137ae41b" containerName="main"
Sep 10 15:59:37.843568 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:37.843520    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="075c5939-6a71-4cc6-87b7-0aa79928b071" containerName="revoke-keys"
Sep 10 15:59:37.846845 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:37.846826    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d6f34-gbgnz"
Sep 10 15:59:37.849367 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:37.849340    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 15:59:37.849508 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:37.849399    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\""
Sep 10 15:59:37.857442 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:37.857412    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d6f34-gbgnz"]
Sep 10 15:59:37.933396 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:37.933356    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f0402f-31b3-4b69-a368-33265a70f92c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5d6f34-gbgnz\" (UID: \"d3f0402f-31b3-4b69-a368-33265a70f92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d6f34-gbgnz"
Sep 10 15:59:38.034173 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:38.034130    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f0402f-31b3-4b69-a368-33265a70f92c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5d6f34-gbgnz\" (UID: \"d3f0402f-31b3-4b69-a368-33265a70f92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d6f34-gbgnz"
Sep 10 15:59:38.034829 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:38.034805    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f0402f-31b3-4b69-a368-33265a70f92c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5d6f34-gbgnz\" (UID: \"d3f0402f-31b3-4b69-a368-33265a70f92c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d6f34-gbgnz"
Sep 10 15:59:38.157806 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:38.157706    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d6f34-gbgnz"
Sep 10 15:59:38.433940 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:38.433848    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d6f34-gbgnz"]
Sep 10 15:59:38.437622 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:59:38.437592    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f0402f_31b3_4b69_a368_33265a70f92c.slice/crio-bc963b670494ef2ffd9cd4530cbf4b3cc97362aae3e08c39ebd47d62caf089fe WatchSource:0}: Error finding container bc963b670494ef2ffd9cd4530cbf4b3cc97362aae3e08c39ebd47d62caf089fe: Status 404 returned error can't find the container with id bc963b670494ef2ffd9cd4530cbf4b3cc97362aae3e08c39ebd47d62caf089fe
Sep 10 15:59:38.461177 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:38.461139    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d6f34-gbgnz" event={"ID":"d3f0402f-31b3-4b69-a368-33265a70f92c","Type":"ContainerStarted","Data":"bc963b670494ef2ffd9cd4530cbf4b3cc97362aae3e08c39ebd47d62caf089fe"}
Sep 10 15:59:39.468725 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:39.468682    2572 generic.go:356] "Generic (PLEG): container finished" podID="d3f0402f-31b3-4b69-a368-33265a70f92c" containerID="d880c27fbf0642e4b5163933cd1d344d4a683b7ba198d3a516788c5ebe2811be" exitCode=0
Sep 10 15:59:39.469133 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:39.468768    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d6f34-gbgnz" event={"ID":"d3f0402f-31b3-4b69-a368-33265a70f92c","Type":"ContainerDied","Data":"d880c27fbf0642e4b5163933cd1d344d4a683b7ba198d3a516788c5ebe2811be"}
Sep 10 15:59:40.613873 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:40.613841    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d6f34-gbgnz"
Sep 10 15:59:40.765222 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:40.765115    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f0402f-31b3-4b69-a368-33265a70f92c-maas-api-service-ca\") pod \"d3f0402f-31b3-4b69-a368-33265a70f92c\" (UID: \"d3f0402f-31b3-4b69-a368-33265a70f92c\") "
Sep 10 15:59:40.765596 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:40.765564    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f0402f-31b3-4b69-a368-33265a70f92c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3f0402f-31b3-4b69-a368-33265a70f92c" (UID: "d3f0402f-31b3-4b69-a368-33265a70f92c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:59:40.866650 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:40.866605    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f0402f-31b3-4b69-a368-33265a70f92c-maas-api-service-ca\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 15:59:41.481147 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:41.481110    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d6f34-gbgnz" event={"ID":"d3f0402f-31b3-4b69-a368-33265a70f92c","Type":"ContainerDied","Data":"bc963b670494ef2ffd9cd4530cbf4b3cc97362aae3e08c39ebd47d62caf089fe"}
Sep 10 15:59:41.481147 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:41.481146    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d6f34-gbgnz"
Sep 10 15:59:41.481417 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:41.481160    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc963b670494ef2ffd9cd4530cbf4b3cc97362aae3e08c39ebd47d62caf089fe"
Sep 10 15:59:49.181446 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:49.181409    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d6f34-gbgnz"]
Sep 10 15:59:49.192827 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:49.192792    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5d6f34-gbgnz"]
Sep 10 15:59:49.992669 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:49.992629    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3f0402f-31b3-4b69-a368-33265a70f92c" path="/var/lib/kubelet/pods/d3f0402f-31b3-4b69-a368-33265a70f92c/volumes"
Sep 10 15:59:55.976865 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:55.976827    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"]
Sep 10 15:59:55.977272 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:55.977121    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq" podUID="11f92f44-7e15-4416-8383-ba28e233d3b3" containerName="istio-proxy" containerID="cri-o://406aef7c0a20146ef807d65a526cb50ae82626bca802974a0054e89c97d7cedf" gracePeriod=30
Sep 10 15:59:56.903682 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:56.903644    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"]
Sep 10 15:59:56.904001 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:56.903988    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3f0402f-31b3-4b69-a368-33265a70f92c" containerName="revoke-keys"
Sep 10 15:59:56.904050 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:56.904002    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3f0402f-31b3-4b69-a368-33265a70f92c" containerName="revoke-keys"
Sep 10 15:59:56.904085 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:56.904072    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3f0402f-31b3-4b69-a368-33265a70f92c" containerName="revoke-keys"
Sep 10 15:59:56.907325 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:56.907294    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:56.909717 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:56.909691    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-50ed73-openshift-default-dockercfg-xj7w2\""
Sep 10 15:59:56.922577 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:56.922550    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"]
Sep 10 15:59:57.007208 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.007170    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-token\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.007208 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.007212    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c87e36-1cc9-4711-8220-e379dbd36f30-istiod-ca-cert\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.007673 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.007235    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x994c\" (UniqueName: \"kubernetes.io/projected/51c87e36-1cc9-4711-8220-e379dbd36f30-kube-api-access-x994c\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.007673 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.007283    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-data\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.007673 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.007361    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-workload-certs\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.007673 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.007408    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-envoy\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.007673 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.007456    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-podinfo\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.007673 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.007507    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-workload-socket\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.007673 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.007562    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-credential-socket\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.109015 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.108965    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-token\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.109196 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.109047    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c87e36-1cc9-4711-8220-e379dbd36f30-istiod-ca-cert\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.109196 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.109086    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x994c\" (UniqueName: \"kubernetes.io/projected/51c87e36-1cc9-4711-8220-e379dbd36f30-kube-api-access-x994c\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.109373 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.109224    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-data\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.109373 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.109259    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-workload-certs\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.109482 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.109377    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-envoy\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.109613 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.109589    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-data\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.109712 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.109690    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-podinfo\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.109780 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.109736    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-workload-socket\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.109780 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.109758    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-workload-certs\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.109885 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.109846    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-credential-socket\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.109934 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.109902    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c87e36-1cc9-4711-8220-e379dbd36f30-istiod-ca-cert\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.110061 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.110006    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-workload-socket\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.110246 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.110226    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-credential-socket\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.111807 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.111785    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-envoy\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.112344 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.112324    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-podinfo\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.117977 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.117945    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-token\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.118083 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.117995    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x994c\" (UniqueName: \"kubernetes.io/projected/51c87e36-1cc9-4711-8220-e379dbd36f30-kube-api-access-x994c\") pod \"e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") " pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.217683 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.217582    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:57.502704 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.502676    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"]
Sep 10 15:59:57.504212 ip-10-0-134-36 kubenswrapper[2572]: W0910 15:59:57.504180    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c87e36_1cc9_4711_8220_e379dbd36f30.slice/crio-f13414d505301a5100bb549c96b38c3477cbe2ccee1786fbaaa53cb403c8cbf8 WatchSource:0}: Error finding container f13414d505301a5100bb549c96b38c3477cbe2ccee1786fbaaa53cb403c8cbf8: Status 404 returned error can't find the container with id f13414d505301a5100bb549c96b38c3477cbe2ccee1786fbaaa53cb403c8cbf8
Sep 10 15:59:57.506526 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.506493    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:59:57.506607 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.506560    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:59:57.506607 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.506589    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:59:57.573714 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.573677    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms" event={"ID":"51c87e36-1cc9-4711-8220-e379dbd36f30","Type":"ContainerStarted","Data":"da8000335c6e84df461cfa56843eeb5af2edf84a4d4dd4502905cbf94506642b"}
Sep 10 15:59:57.573910 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.573739    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms" event={"ID":"51c87e36-1cc9-4711-8220-e379dbd36f30","Type":"ContainerStarted","Data":"f13414d505301a5100bb549c96b38c3477cbe2ccee1786fbaaa53cb403c8cbf8"}
Sep 10 15:59:57.602627 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:57.602555    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms" podStartSLOduration=1.602528363 podStartE2EDuration="1.602528363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:59:57.599418867 +0000 UTC m=+1070.271828952" watchObservedRunningTime="2026-09-10 15:59:57.602528363 +0000 UTC m=+1070.274938447"
Sep 10 15:59:58.218164 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:58.218124    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:58.223242 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:58.223214    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:58.580135 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:58.580107    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 15:59:58.581105 ip-10-0-134-36 kubenswrapper[2572]: I0910 15:59:58.581081    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 16:00:00.160741 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:00.160656    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817600-j8nrh"]
Sep 10 16:00:00.164252 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:00.164231    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-j8nrh"
Sep 10 16:00:00.167462 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:00.167434    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\""
Sep 10 16:00:00.174047 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:00.174026    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-j8nrh"
Sep 10 16:00:00.182361 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:00.182330    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817600-j8nrh"]
Sep 10 16:00:00.456045 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:00.456006    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817600-j8nrh"]
Sep 10 16:00:00.457128 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:00:00.457101    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f90761a_4e2a_473a_bc4b_1f2eda9dec27.slice/crio-340cbcdfbf266f064a7f672a6563af54b67c20ff47545e0f957fb25bf2b29242 WatchSource:0}: Error finding container 340cbcdfbf266f064a7f672a6563af54b67c20ff47545e0f957fb25bf2b29242: Status 404 returned error can't find the container with id 340cbcdfbf266f064a7f672a6563af54b67c20ff47545e0f957fb25bf2b29242
Sep 10 16:00:00.592183 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:00.592141    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-j8nrh" event={"ID":"9f90761a-4e2a-473a-bc4b-1f2eda9dec27","Type":"ContainerStarted","Data":"340cbcdfbf266f064a7f672a6563af54b67c20ff47545e0f957fb25bf2b29242"}
Sep 10 16:00:00.619043 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:00.619008    2572 patch_prober.go:28] interesting pod/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 16:00:00.619223 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:00.619066    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq" podUID="11f92f44-7e15-4416-8383-ba28e233d3b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 16:00:01.227601 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.227566    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 16:00:01.349615 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.349583    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf25b\" (UniqueName: \"kubernetes.io/projected/11f92f44-7e15-4416-8383-ba28e233d3b3-kube-api-access-lf25b\") pod \"11f92f44-7e15-4416-8383-ba28e233d3b3\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") "
Sep 10 16:00:01.349801 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.349631    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-data\") pod \"11f92f44-7e15-4416-8383-ba28e233d3b3\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") "
Sep 10 16:00:01.349801 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.349655    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-podinfo\") pod \"11f92f44-7e15-4416-8383-ba28e233d3b3\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") "
Sep 10 16:00:01.349801 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.349698    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-workload-socket\") pod \"11f92f44-7e15-4416-8383-ba28e233d3b3\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") "
Sep 10 16:00:01.349801 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.349742    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-envoy\") pod \"11f92f44-7e15-4416-8383-ba28e233d3b3\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") "
Sep 10 16:00:01.349801 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.349767    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-token\") pod \"11f92f44-7e15-4416-8383-ba28e233d3b3\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") "
Sep 10 16:00:01.350122 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.349821    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f92f44-7e15-4416-8383-ba28e233d3b3-istiod-ca-cert\") pod \"11f92f44-7e15-4416-8383-ba28e233d3b3\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") "
Sep 10 16:00:01.350122 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.349857    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-workload-certs\") pod \"11f92f44-7e15-4416-8383-ba28e233d3b3\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") "
Sep 10 16:00:01.350122 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.349883    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-credential-socket\") pod \"11f92f44-7e15-4416-8383-ba28e233d3b3\" (UID: \"11f92f44-7e15-4416-8383-ba28e233d3b3\") "
Sep 10 16:00:01.350122 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.349916    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "11f92f44-7e15-4416-8383-ba28e233d3b3" (UID: "11f92f44-7e15-4416-8383-ba28e233d3b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:00:01.350122 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.350108    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "11f92f44-7e15-4416-8383-ba28e233d3b3" (UID: "11f92f44-7e15-4416-8383-ba28e233d3b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:00:01.350423 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.350154    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-data\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:00:01.350423 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.350298    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "11f92f44-7e15-4416-8383-ba28e233d3b3" (UID: "11f92f44-7e15-4416-8383-ba28e233d3b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:00:01.350423 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.350349    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f92f44-7e15-4416-8383-ba28e233d3b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "11f92f44-7e15-4416-8383-ba28e233d3b3" (UID: "11f92f44-7e15-4416-8383-ba28e233d3b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:00:01.350533 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.350508    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "11f92f44-7e15-4416-8383-ba28e233d3b3" (UID: "11f92f44-7e15-4416-8383-ba28e233d3b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:00:01.352063 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.352034    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "11f92f44-7e15-4416-8383-ba28e233d3b3" (UID: "11f92f44-7e15-4416-8383-ba28e233d3b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:00:01.352180 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.352083    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "11f92f44-7e15-4416-8383-ba28e233d3b3" (UID: "11f92f44-7e15-4416-8383-ba28e233d3b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:00:01.352370 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.352352    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "11f92f44-7e15-4416-8383-ba28e233d3b3" (UID: "11f92f44-7e15-4416-8383-ba28e233d3b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:00:01.352370 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.352363    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f92f44-7e15-4416-8383-ba28e233d3b3-kube-api-access-lf25b" (OuterVolumeSpecName: "kube-api-access-lf25b") pod "11f92f44-7e15-4416-8383-ba28e233d3b3" (UID: "11f92f44-7e15-4416-8383-ba28e233d3b3"). InnerVolumeSpecName "kube-api-access-lf25b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:00:01.451090 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.451052    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f92f44-7e15-4416-8383-ba28e233d3b3-istiod-ca-cert\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:00:01.451090 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.451082    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-workload-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:00:01.451090 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.451093    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-credential-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:00:01.451364 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.451103    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf25b\" (UniqueName: \"kubernetes.io/projected/11f92f44-7e15-4416-8383-ba28e233d3b3-kube-api-access-lf25b\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:00:01.451364 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.451112    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-podinfo\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:00:01.451364 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.451120    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-workload-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:00:01.451364 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.451129    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-envoy\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:00:01.451364 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.451136    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f92f44-7e15-4416-8383-ba28e233d3b3-istio-token\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:00:01.599150 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.599112    2572 generic.go:356] "Generic (PLEG): container finished" podID="9f90761a-4e2a-473a-bc4b-1f2eda9dec27" containerID="57847dedf4d3a4fea6e8707a714e2b6458ddd9b8c51433cf07e40194c8e2bcc0" exitCode=0
Sep 10 16:00:01.599364 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.599158    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-j8nrh" event={"ID":"9f90761a-4e2a-473a-bc4b-1f2eda9dec27","Type":"ContainerDied","Data":"57847dedf4d3a4fea6e8707a714e2b6458ddd9b8c51433cf07e40194c8e2bcc0"}
Sep 10 16:00:01.601014 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.600953    2572 generic.go:356] "Generic (PLEG): container finished" podID="11f92f44-7e15-4416-8383-ba28e233d3b3" containerID="406aef7c0a20146ef807d65a526cb50ae82626bca802974a0054e89c97d7cedf" exitCode=0
Sep 10 16:00:01.601014 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.600996    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"
Sep 10 16:00:01.601151 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.601017    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq" event={"ID":"11f92f44-7e15-4416-8383-ba28e233d3b3","Type":"ContainerDied","Data":"406aef7c0a20146ef807d65a526cb50ae82626bca802974a0054e89c97d7cedf"}
Sep 10 16:00:01.601151 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.601059    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq" event={"ID":"11f92f44-7e15-4416-8383-ba28e233d3b3","Type":"ContainerDied","Data":"cf57b73756bf4418aefd4ea8d409fecf47023b182826b7f3a87ec6a7e4ff759f"}
Sep 10 16:00:01.601151 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.601083    2572 scope.go:117] "RemoveContainer" containerID="406aef7c0a20146ef807d65a526cb50ae82626bca802974a0054e89c97d7cedf"
Sep 10 16:00:01.610601 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.610580    2572 scope.go:117] "RemoveContainer" containerID="0b93d939af3952b212bce7e917f3727b3a2df12e1579743a3cb5ffe1398760d6"
Sep 10 16:00:01.620760 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.620735    2572 scope.go:117] "RemoveContainer" containerID="406aef7c0a20146ef807d65a526cb50ae82626bca802974a0054e89c97d7cedf"
Sep 10 16:00:01.621105 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:00:01.621085    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406aef7c0a20146ef807d65a526cb50ae82626bca802974a0054e89c97d7cedf\": container with ID starting with 406aef7c0a20146ef807d65a526cb50ae82626bca802974a0054e89c97d7cedf not found: ID does not exist" containerID="406aef7c0a20146ef807d65a526cb50ae82626bca802974a0054e89c97d7cedf"
Sep 10 16:00:01.621195 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.621113    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406aef7c0a20146ef807d65a526cb50ae82626bca802974a0054e89c97d7cedf"} err="failed to get container status \"406aef7c0a20146ef807d65a526cb50ae82626bca802974a0054e89c97d7cedf\": rpc error: code = NotFound desc = could not find container \"406aef7c0a20146ef807d65a526cb50ae82626bca802974a0054e89c97d7cedf\": container with ID starting with 406aef7c0a20146ef807d65a526cb50ae82626bca802974a0054e89c97d7cedf not found: ID does not exist"
Sep 10 16:00:01.621195 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.621129    2572 scope.go:117] "RemoveContainer" containerID="0b93d939af3952b212bce7e917f3727b3a2df12e1579743a3cb5ffe1398760d6"
Sep 10 16:00:01.621522 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:00:01.621502    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b93d939af3952b212bce7e917f3727b3a2df12e1579743a3cb5ffe1398760d6\": container with ID starting with 0b93d939af3952b212bce7e917f3727b3a2df12e1579743a3cb5ffe1398760d6 not found: ID does not exist" containerID="0b93d939af3952b212bce7e917f3727b3a2df12e1579743a3cb5ffe1398760d6"
Sep 10 16:00:01.621615 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.621527    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b93d939af3952b212bce7e917f3727b3a2df12e1579743a3cb5ffe1398760d6"} err="failed to get container status \"0b93d939af3952b212bce7e917f3727b3a2df12e1579743a3cb5ffe1398760d6\": rpc error: code = NotFound desc = could not find container \"0b93d939af3952b212bce7e917f3727b3a2df12e1579743a3cb5ffe1398760d6\": container with ID starting with 0b93d939af3952b212bce7e917f3727b3a2df12e1579743a3cb5ffe1398760d6 not found: ID does not exist"
Sep 10 16:00:01.633793 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.633765    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"]
Sep 10 16:00:01.638475 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.638447    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5d6f34-openshift-default-5978845b5b-6jlnq"]
Sep 10 16:00:01.993103 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:01.993010    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11f92f44-7e15-4416-8383-ba28e233d3b3" path="/var/lib/kubelet/pods/11f92f44-7e15-4416-8383-ba28e233d3b3/volumes"
Sep 10 16:00:02.741712 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:02.741675    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-j8nrh"
Sep 10 16:00:03.615782 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:03.615750    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-j8nrh"
Sep 10 16:00:03.615782 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:03.615777    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-j8nrh" event={"ID":"9f90761a-4e2a-473a-bc4b-1f2eda9dec27","Type":"ContainerDied","Data":"340cbcdfbf266f064a7f672a6563af54b67c20ff47545e0f957fb25bf2b29242"}
Sep 10 16:00:03.616013 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:03.615823    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="340cbcdfbf266f064a7f672a6563af54b67c20ff47545e0f957fb25bf2b29242"
Sep 10 16:00:09.906479 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:09.906444    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-678f625f-2mrjp"]
Sep 10 16:00:09.906950 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:09.906779    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f90761a-4e2a-473a-bc4b-1f2eda9dec27" containerName="cleanup"
Sep 10 16:00:09.906950 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:09.906791    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f90761a-4e2a-473a-bc4b-1f2eda9dec27" containerName="cleanup"
Sep 10 16:00:09.906950 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:09.906805    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f92f44-7e15-4416-8383-ba28e233d3b3" containerName="istio-proxy"
Sep 10 16:00:09.906950 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:09.906811    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f92f44-7e15-4416-8383-ba28e233d3b3" containerName="istio-proxy"
Sep 10 16:00:09.906950 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:09.906872    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="11f92f44-7e15-4416-8383-ba28e233d3b3" containerName="istio-proxy"
Sep 10 16:00:09.906950 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:09.906883    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f90761a-4e2a-473a-bc4b-1f2eda9dec27" containerName="cleanup"
Sep 10 16:00:09.911405 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:09.911385    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-678f625f-2mrjp"
Sep 10 16:00:09.914066 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:09.914048    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:00:09.914140 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:09.914062    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\""
Sep 10 16:00:09.924397 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:09.924064    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-678f625f-2mrjp"]
Sep 10 16:00:10.053585 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:10.053548    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b7e30c-ad13-4c6c-bae9-1c6969508737-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-678f625f-2mrjp\" (UID: \"04b7e30c-ad13-4c6c-bae9-1c6969508737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-678f625f-2mrjp"
Sep 10 16:00:10.154848 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:10.154814    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b7e30c-ad13-4c6c-bae9-1c6969508737-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-678f625f-2mrjp\" (UID: \"04b7e30c-ad13-4c6c-bae9-1c6969508737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-678f625f-2mrjp"
Sep 10 16:00:10.155690 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:10.155657    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b7e30c-ad13-4c6c-bae9-1c6969508737-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-678f625f-2mrjp\" (UID: \"04b7e30c-ad13-4c6c-bae9-1c6969508737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-678f625f-2mrjp"
Sep 10 16:00:10.229373 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:10.229256    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-678f625f-2mrjp"
Sep 10 16:00:10.515991 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:10.515935    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-678f625f-2mrjp"]
Sep 10 16:00:10.664396 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:10.664355    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-678f625f-2mrjp" event={"ID":"04b7e30c-ad13-4c6c-bae9-1c6969508737","Type":"ContainerStarted","Data":"b0fde2420e2ffb892c56a14612c2d3e11308b8a05232e31d3cc7c80fcff09b33"}
Sep 10 16:00:11.672046 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:11.672011    2572 generic.go:356] "Generic (PLEG): container finished" podID="04b7e30c-ad13-4c6c-bae9-1c6969508737" containerID="afb940e28acd481d40caaded52455005cce86e2921b393eae9e08805787745e7" exitCode=0
Sep 10 16:00:11.672444 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:11.672058    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-678f625f-2mrjp" event={"ID":"04b7e30c-ad13-4c6c-bae9-1c6969508737","Type":"ContainerDied","Data":"afb940e28acd481d40caaded52455005cce86e2921b393eae9e08805787745e7"}
Sep 10 16:00:12.817000 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:12.816967    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-678f625f-2mrjp"
Sep 10 16:00:12.979638 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:12.979600    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b7e30c-ad13-4c6c-bae9-1c6969508737-maas-api-service-ca\") pod \"04b7e30c-ad13-4c6c-bae9-1c6969508737\" (UID: \"04b7e30c-ad13-4c6c-bae9-1c6969508737\") "
Sep 10 16:00:12.980002 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:12.979974    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b7e30c-ad13-4c6c-bae9-1c6969508737-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04b7e30c-ad13-4c6c-bae9-1c6969508737" (UID: "04b7e30c-ad13-4c6c-bae9-1c6969508737"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:00:13.081195 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:13.081157    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b7e30c-ad13-4c6c-bae9-1c6969508737-maas-api-service-ca\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:00:13.684680 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:13.684643    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-678f625f-2mrjp"
Sep 10 16:00:13.684868 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:13.684681    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-678f625f-2mrjp" event={"ID":"04b7e30c-ad13-4c6c-bae9-1c6969508737","Type":"ContainerDied","Data":"b0fde2420e2ffb892c56a14612c2d3e11308b8a05232e31d3cc7c80fcff09b33"}
Sep 10 16:00:13.684868 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:13.684738    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0fde2420e2ffb892c56a14612c2d3e11308b8a05232e31d3cc7c80fcff09b33"
Sep 10 16:00:14.622499 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:14.622451    2572 patch_prober.go:28] interesting pod/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 16:00:14.622907 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:14.622512    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms" podUID="51c87e36-1cc9-4711-8220-e379dbd36f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 16:00:16.704254 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:16.704221    2572 generic.go:356] "Generic (PLEG): container finished" podID="51c87e36-1cc9-4711-8220-e379dbd36f30" containerID="da8000335c6e84df461cfa56843eeb5af2edf84a4d4dd4502905cbf94506642b" exitCode=0
Sep 10 16:00:16.704701 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:16.704295    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms" event={"ID":"51c87e36-1cc9-4711-8220-e379dbd36f30","Type":"ContainerDied","Data":"da8000335c6e84df461cfa56843eeb5af2edf84a4d4dd4502905cbf94506642b"}
Sep 10 16:00:16.704760 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:16.704714    2572 scope.go:117] "RemoveContainer" containerID="da8000335c6e84df461cfa56843eeb5af2edf84a4d4dd4502905cbf94506642b"
Sep 10 16:00:16.705816 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:16.705787    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:00:16.705888 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:16.705874    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:00:16.705935 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:16.705914    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:00:17.711761 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:17.711725    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms" event={"ID":"51c87e36-1cc9-4711-8220-e379dbd36f30","Type":"ContainerStarted","Data":"2fb117b983e26a1ef412a7ce7355329e127732f972ad84dadad6f53ebddbb404"}
Sep 10 16:00:18.218135 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:18.218093    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 16:00:18.218558 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:18.218528    2572 patch_prober.go:28] interesting pod/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 16:00:18.218785 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:18.218754    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms" podUID="51c87e36-1cc9-4711-8220-e379dbd36f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 16:00:19.218613 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:19.218575    2572 patch_prober.go:28] interesting pod/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 16:00:19.219215 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:19.219178    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms" podUID="51c87e36-1cc9-4711-8220-e379dbd36f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 16:00:20.218897 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:20.218858    2572 patch_prober.go:28] interesting pod/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 16:00:20.219546 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:20.219508    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms" podUID="51c87e36-1cc9-4711-8220-e379dbd36f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 16:00:20.829397 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:20.829355    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-678f625f-2mrjp"]
Sep 10 16:00:20.831333 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:20.831284    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-678f625f-2mrjp"]
Sep 10 16:00:21.218855 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:21.218743    2572 patch_prober.go:28] interesting pod/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 16:00:21.218855 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:21.218812    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms" podUID="51c87e36-1cc9-4711-8220-e379dbd36f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 16:00:21.992214 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:21.992176    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04b7e30c-ad13-4c6c-bae9-1c6969508737" path="/var/lib/kubelet/pods/04b7e30c-ad13-4c6c-bae9-1c6969508737/volumes"
Sep 10 16:00:22.218875 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:22.218831    2572 patch_prober.go:28] interesting pod/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 16:00:22.219192 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:22.219163    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms" podUID="51c87e36-1cc9-4711-8220-e379dbd36f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 16:00:23.222856 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:23.222825    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 16:00:23.223260 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:23.223185    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 16:00:23.223851 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:23.223830    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 16:00:25.328219 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.328180    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"]
Sep 10 16:00:25.328732 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.328713    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f92f44-7e15-4416-8383-ba28e233d3b3" containerName="istio-proxy"
Sep 10 16:00:25.328779 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.328736    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f92f44-7e15-4416-8383-ba28e233d3b3" containerName="istio-proxy"
Sep 10 16:00:25.328779 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.328750    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b7e30c-ad13-4c6c-bae9-1c6969508737" containerName="revoke-keys"
Sep 10 16:00:25.328779 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.328759    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b7e30c-ad13-4c6c-bae9-1c6969508737" containerName="revoke-keys"
Sep 10 16:00:25.328870 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.328852    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="11f92f44-7e15-4416-8383-ba28e233d3b3" containerName="istio-proxy"
Sep 10 16:00:25.328870 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.328867    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="04b7e30c-ad13-4c6c-bae9-1c6969508737" containerName="revoke-keys"
Sep 10 16:00:25.331187 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.331166    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.335895 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.335875    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-67df2b-openshift-default-dockercfg-5glq8\""
Sep 10 16:00:25.388280 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.388237    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-istio-data\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.388280 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.388280    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111222db-994b-41f5-a88d-56876c84d47e-istio-token\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.388521 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.388339    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-workload-socket\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.388521 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.388382    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111222db-994b-41f5-a88d-56876c84d47e-istiod-ca-cert\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.388521 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.388417    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbkzv\" (UniqueName: \"kubernetes.io/projected/111222db-994b-41f5-a88d-56876c84d47e-kube-api-access-qbkzv\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.388521 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.388459    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-istio-envoy\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.388666 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.388530    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-workload-certs\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.388666 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.388555    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111222db-994b-41f5-a88d-56876c84d47e-istio-podinfo\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.388666 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.388588    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-credential-socket\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.405319 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.405275    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"]
Sep 10 16:00:25.489882 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.489839    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-workload-certs\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.490092 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.489890    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111222db-994b-41f5-a88d-56876c84d47e-istio-podinfo\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.490092 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.490076    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-credential-socket\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.490222 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.490136    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-istio-data\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.490222 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.490166    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111222db-994b-41f5-a88d-56876c84d47e-istio-token\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.490222 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.490211    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-workload-socket\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.490430 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.490240    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111222db-994b-41f5-a88d-56876c84d47e-istiod-ca-cert\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.490430 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.490287    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbkzv\" (UniqueName: \"kubernetes.io/projected/111222db-994b-41f5-a88d-56876c84d47e-kube-api-access-qbkzv\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.490430 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.490343    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-workload-certs\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.490430 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.490344    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-istio-envoy\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.490665 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.490451    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-credential-socket\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.490665 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.490520    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-istio-data\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.490665 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.490545    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-workload-socket\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.490999 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.490977    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111222db-994b-41f5-a88d-56876c84d47e-istiod-ca-cert\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.492470 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.492443    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-istio-envoy\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.492574 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.492554    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111222db-994b-41f5-a88d-56876c84d47e-istio-podinfo\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.499905 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.499875    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbkzv\" (UniqueName: \"kubernetes.io/projected/111222db-994b-41f5-a88d-56876c84d47e-kube-api-access-qbkzv\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.500092 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.500073    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111222db-994b-41f5-a88d-56876c84d47e-istio-token\") pod \"e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") " pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.641553 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.641452    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:25.921472 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.921440    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"]
Sep 10 16:00:25.923744 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:00:25.923714    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111222db_994b_41f5_a88d_56876c84d47e.slice/crio-35538465c7b95d73ea69d7b831649ab110687d19c37b41fbed09ca145717917e WatchSource:0}: Error finding container 35538465c7b95d73ea69d7b831649ab110687d19c37b41fbed09ca145717917e: Status 404 returned error can't find the container with id 35538465c7b95d73ea69d7b831649ab110687d19c37b41fbed09ca145717917e
Sep 10 16:00:25.926249 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.926215    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:00:25.926450 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.926326    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:00:25.926450 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:25.926380    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:00:26.771711 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:26.771671    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt" event={"ID":"111222db-994b-41f5-a88d-56876c84d47e","Type":"ContainerStarted","Data":"24c4fed295eeaaf803cf0e707775e2eeb1063000988b025e2046f7bdd021826a"}
Sep 10 16:00:26.772183 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:26.771739    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt" event={"ID":"111222db-994b-41f5-a88d-56876c84d47e","Type":"ContainerStarted","Data":"35538465c7b95d73ea69d7b831649ab110687d19c37b41fbed09ca145717917e"}
Sep 10 16:00:26.799090 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:26.799025    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt" podStartSLOduration=1.799008999 podStartE2EDuration="1.799008999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:26.79669677 +0000 UTC m=+1099.469106852" watchObservedRunningTime="2026-09-10 16:00:26.799008999 +0000 UTC m=+1099.471419082"
Sep 10 16:00:27.642356 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:27.642286    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:27.647526 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:27.647496    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:27.777082 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:27.777047    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:27.778375 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:27.778354    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:43.818547 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:43.818492    2572 patch_prober.go:28] interesting pod/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 16:00:43.819159 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:43.818584    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt" podUID="111222db-994b-41f5-a88d-56876c84d47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 16:00:44.887769 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:44.887730    2572 generic.go:356] "Generic (PLEG): container finished" podID="111222db-994b-41f5-a88d-56876c84d47e" containerID="24c4fed295eeaaf803cf0e707775e2eeb1063000988b025e2046f7bdd021826a" exitCode=0
Sep 10 16:00:44.888206 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:44.887805    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt" event={"ID":"111222db-994b-41f5-a88d-56876c84d47e","Type":"ContainerDied","Data":"24c4fed295eeaaf803cf0e707775e2eeb1063000988b025e2046f7bdd021826a"}
Sep 10 16:00:44.888274 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:44.888259    2572 scope.go:117] "RemoveContainer" containerID="24c4fed295eeaaf803cf0e707775e2eeb1063000988b025e2046f7bdd021826a"
Sep 10 16:00:44.889410 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:44.889377    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:00:44.889518 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:44.889451    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:00:44.889518 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:44.889482    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:00:45.895870 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:45.895836    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt" event={"ID":"111222db-994b-41f5-a88d-56876c84d47e","Type":"ContainerStarted","Data":"09afb2dcd3dcfe607a3c04ada982bef50b4f80e961ead5fbcedae29adc211022"}
Sep 10 16:00:46.642660 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:46.642600    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:46.642873 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:46.642732    2572 patch_prober.go:28] interesting pod/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 10 16:00:46.642873 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:46.642801    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt" podUID="111222db-994b-41f5-a88d-56876c84d47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 10 16:00:47.643032 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:47.642991    2572 patch_prober.go:28] interesting pod/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 10 16:00:47.644340 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:47.643611    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt" podUID="111222db-994b-41f5-a88d-56876c84d47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 10 16:00:48.642369 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:48.642323    2572 patch_prober.go:28] interesting pod/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 10 16:00:48.642557 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:48.642392    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt" podUID="111222db-994b-41f5-a88d-56876c84d47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 10 16:00:49.642705 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:49.642654    2572 patch_prober.go:28] interesting pod/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 10 16:00:49.643377 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:49.643336    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt" podUID="111222db-994b-41f5-a88d-56876c84d47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 10 16:00:50.642726 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:50.642674    2572 patch_prober.go:28] interesting pod/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 10 16:00:50.643128 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:50.642749    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt" podUID="111222db-994b-41f5-a88d-56876c84d47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 10 16:00:51.865592 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:51.865556    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:51.866186 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:51.866038    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:00:51.867106 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:00:51.867084    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:01:11.558341 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.558270    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"]
Sep 10 16:01:11.562049 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.562026    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.564819 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.564794    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-33451c-openshift-default-dockercfg-mgkjf\""
Sep 10 16:01:11.583644 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.583608    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"]
Sep 10 16:01:11.623424 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.623389    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-podinfo\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.623424 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.623427    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istiod-ca-cert\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.623705 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.623456    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-workload-socket\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.623705 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.623497    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-envoy\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.623705 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.623548    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-data\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.623705 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.623592    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-token\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.623705 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.623648    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-credential-socket\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.623705 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.623670    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-workload-certs\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.623705 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.623689    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxhxd\" (UniqueName: \"kubernetes.io/projected/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-kube-api-access-wxhxd\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.724847 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.724802    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-data\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.724847 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.724847    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-token\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.725096 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.724885    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-credential-socket\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.725096 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.724904    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-workload-certs\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.725096 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.724924    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxhxd\" (UniqueName: \"kubernetes.io/projected/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-kube-api-access-wxhxd\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.725096 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.724947    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-podinfo\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.725096 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.724967    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istiod-ca-cert\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.725096 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.725000    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-workload-socket\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.725096 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.725035    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-envoy\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.725508 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.725296    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-data\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.725508 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.725385    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-workload-certs\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.725508 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.725458    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-workload-socket\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.725686 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.725544    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-credential-socket\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.725799 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.725776    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istiod-ca-cert\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.727673 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.727647    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-podinfo\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.727673 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.727664    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-envoy\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.733745 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.733718    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-token\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.734444 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.734419    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxhxd\" (UniqueName: \"kubernetes.io/projected/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-kube-api-access-wxhxd\") pod \"e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") " pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:11.872371 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:11.872267    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:12.165857 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:01:12.165816    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2997d9f3_34a3_47b5_86dd_e4bb79bb0cd9.slice/crio-c21967c727aae3627448f63ae68db2bd6d98d465b85913778392f7ba221cea6a WatchSource:0}: Error finding container c21967c727aae3627448f63ae68db2bd6d98d465b85913778392f7ba221cea6a: Status 404 returned error can't find the container with id c21967c727aae3627448f63ae68db2bd6d98d465b85913778392f7ba221cea6a
Sep 10 16:01:12.168572 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:12.168532    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:01:12.168685 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:12.168611    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:01:12.168685 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:12.168646    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:01:12.171537 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:12.171510    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"]
Sep 10 16:01:13.052259 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:13.052221    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n" event={"ID":"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9","Type":"ContainerStarted","Data":"ed6852e1fd45ed592a4240a4e8ff1dda5aad7f8c7f0874192680774ee0c3aca6"}
Sep 10 16:01:13.052678 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:13.052282    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n" event={"ID":"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9","Type":"ContainerStarted","Data":"c21967c727aae3627448f63ae68db2bd6d98d465b85913778392f7ba221cea6a"}
Sep 10 16:01:13.075916 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:13.075847    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n" podStartSLOduration=2.075826591 podStartE2EDuration="2.075826591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:13.075289615 +0000 UTC m=+1145.747699711" watchObservedRunningTime="2026-09-10 16:01:13.075826591 +0000 UTC m=+1145.748236679"
Sep 10 16:01:13.872584 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:13.872546    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:13.878233 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:13.878206    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:14.057945 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:14.057916    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:14.059103 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:14.059080    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:19.041515 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.041477    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"]
Sep 10 16:01:19.045347 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.045328    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.048646 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.048609    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-50ed73\"/\"kube-root-ca.crt\""
Sep 10 16:01:19.048646 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.048624    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-50ed73\"/\"auth-test-model-50ed73-kserve-self-signed-certs\""
Sep 10 16:01:19.048853 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.048626    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-50ed73\"/\"default-dockercfg-5qjdv\""
Sep 10 16:01:19.048853 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.048694    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-50ed73\"/\"openshift-service-ca.crt\""
Sep 10 16:01:19.054414 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.054387    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"]
Sep 10 16:01:19.094548 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.094513    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tscnm\" (UniqueName: \"kubernetes.io/projected/227622f8-939d-4ff5-a22f-4662135c4ccf-kube-api-access-tscnm\") pod \"auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.094548 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.094555    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-home\") pod \"auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.094788 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.094661    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-tmp-dir\") pod \"auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.094788 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.094715    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-model-cache\") pod \"auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.094788 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.094768    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-dshm\") pod \"auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.094953 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.094805    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/227622f8-939d-4ff5-a22f-4662135c4ccf-tls-certs\") pod \"auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.195730 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.195690    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tscnm\" (UniqueName: \"kubernetes.io/projected/227622f8-939d-4ff5-a22f-4662135c4ccf-kube-api-access-tscnm\") pod \"auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.195730 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.195732    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-home\") pod \"auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.196006 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.195861    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-tmp-dir\") pod \"auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.196006 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.195911    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-model-cache\") pod \"auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.196006 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.195971    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-dshm\") pod \"auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.196154 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.196011    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/227622f8-939d-4ff5-a22f-4662135c4ccf-tls-certs\") pod \"auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.196154 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.196143    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-home\") pod \"auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.196224 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.196177    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-tmp-dir\") pod \"auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.196294 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.196277    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-model-cache\") pod \"auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.198283 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.198246    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-dshm\") pod \"auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.198561 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.198543    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/227622f8-939d-4ff5-a22f-4662135c4ccf-tls-certs\") pod \"auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.204336 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.204315    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tscnm\" (UniqueName: \"kubernetes.io/projected/227622f8-939d-4ff5-a22f-4662135c4ccf-kube-api-access-tscnm\") pod \"auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.355636 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.355604    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:19.630799 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:19.630769    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"]
Sep 10 16:01:19.632354 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:01:19.632322    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod227622f8_939d_4ff5_a22f_4662135c4ccf.slice/crio-14b084b9cae173597bfbe3f8ecb24317957a450923689608cdf76ed2d1680bcf WatchSource:0}: Error finding container 14b084b9cae173597bfbe3f8ecb24317957a450923689608cdf76ed2d1680bcf: Status 404 returned error can't find the container with id 14b084b9cae173597bfbe3f8ecb24317957a450923689608cdf76ed2d1680bcf
Sep 10 16:01:20.092196 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:20.092154    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x" event={"ID":"227622f8-939d-4ff5-a22f-4662135c4ccf","Type":"ContainerStarted","Data":"1ea403e3308d2130f43804e80871ba670f723e7a01341f23ffd3ca2ecc35e923"}
Sep 10 16:01:20.092625 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:20.092205    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x" event={"ID":"227622f8-939d-4ff5-a22f-4662135c4ccf","Type":"ContainerStarted","Data":"14b084b9cae173597bfbe3f8ecb24317957a450923689608cdf76ed2d1680bcf"}
Sep 10 16:01:20.110484 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:20.110427    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x" podStartSLOduration=1.110412394 podStartE2EDuration="1.110412394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:20.109404212 +0000 UTC m=+1152.781814295" watchObservedRunningTime="2026-09-10 16:01:20.110412394 +0000 UTC m=+1152.782822476"
Sep 10 16:01:29.356804 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:29.356760    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:29.357320 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:29.356822    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:29.370400 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:29.370365    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:30.099607 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:30.099505    2572 patch_prober.go:28] interesting pod/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 16:01:30.099607 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:30.099573    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n" podUID="2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 16:01:30.175455 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:30.175421    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:01:32.177605 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:32.177512    2572 generic.go:356] "Generic (PLEG): container finished" podID="2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" containerID="ed6852e1fd45ed592a4240a4e8ff1dda5aad7f8c7f0874192680774ee0c3aca6" exitCode=0
Sep 10 16:01:32.177605 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:32.177588    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n" event={"ID":"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9","Type":"ContainerDied","Data":"ed6852e1fd45ed592a4240a4e8ff1dda5aad7f8c7f0874192680774ee0c3aca6"}
Sep 10 16:01:32.178052 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:32.178013    2572 scope.go:117] "RemoveContainer" containerID="ed6852e1fd45ed592a4240a4e8ff1dda5aad7f8c7f0874192680774ee0c3aca6"
Sep 10 16:01:32.179361 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:32.179325    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:01:32.179443 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:32.179405    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:01:32.179443 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:32.179440    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:01:33.185089 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:33.185057    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n" event={"ID":"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9","Type":"ContainerStarted","Data":"c5d8c0b34cdf240afc4ce3aa97c217ac5387943e566a942f919e0c3e632a40ba"}
Sep 10 16:01:33.873352 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:33.873292    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:33.873650 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:33.873625    2572 patch_prober.go:28] interesting pod/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 10 16:01:33.873753 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:33.873698    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n" podUID="2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 10 16:01:34.873055 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:34.873016    2572 patch_prober.go:28] interesting pod/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 10 16:01:34.873507 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:34.873073    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n" podUID="2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 10 16:01:35.873634 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:35.873591    2572 patch_prober.go:28] interesting pod/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 10 16:01:35.874069 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:35.873653    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n" podUID="2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 10 16:01:36.873028 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:36.872989    2572 patch_prober.go:28] interesting pod/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 10 16:01:36.873217 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:36.873046    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n" podUID="2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 10 16:01:37.873492 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:37.873450    2572 patch_prober.go:28] interesting pod/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 10 16:01:37.873915 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:37.873515    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n" podUID="2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 10 16:01:38.876711 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:38.876673    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:38.877165 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:38.876955    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:38.877695 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:38.877677    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:01:40.803172 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:40.803128    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7592-aba28b"]
Sep 10 16:01:40.806527 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:40.806507    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7592-aba28b"
Sep 10 16:01:40.809038 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:40.809015    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-j2pv4\""
Sep 10 16:01:40.813457 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:40.813431    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7592-aba28b"]
Sep 10 16:01:40.915461 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:40.915409    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9kjq\" (UniqueName: \"kubernetes.io/projected/6e1ce14f-bfda-4b33-9aff-fecc89a2f42a-kube-api-access-h9kjq\") pod \"test-curl-7592-aba28b\" (UID: \"6e1ce14f-bfda-4b33-9aff-fecc89a2f42a\") " pod="opendatahub/test-curl-7592-aba28b"
Sep 10 16:01:41.016940 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:41.016902    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9kjq\" (UniqueName: \"kubernetes.io/projected/6e1ce14f-bfda-4b33-9aff-fecc89a2f42a-kube-api-access-h9kjq\") pod \"test-curl-7592-aba28b\" (UID: \"6e1ce14f-bfda-4b33-9aff-fecc89a2f42a\") " pod="opendatahub/test-curl-7592-aba28b"
Sep 10 16:01:41.026450 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:41.026408    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9kjq\" (UniqueName: \"kubernetes.io/projected/6e1ce14f-bfda-4b33-9aff-fecc89a2f42a-kube-api-access-h9kjq\") pod \"test-curl-7592-aba28b\" (UID: \"6e1ce14f-bfda-4b33-9aff-fecc89a2f42a\") " pod="opendatahub/test-curl-7592-aba28b"
Sep 10 16:01:41.117856 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:41.117761    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7592-aba28b"
Sep 10 16:01:41.391991 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:41.391962    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7592-aba28b"]
Sep 10 16:01:41.393775 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:01:41.393742    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e1ce14f_bfda_4b33_9aff_fecc89a2f42a.slice/crio-990847e614f5050c076c34d0085a166c59c11d2b8c087b876d008f8ef34df453 WatchSource:0}: Error finding container 990847e614f5050c076c34d0085a166c59c11d2b8c087b876d008f8ef34df453: Status 404 returned error can't find the container with id 990847e614f5050c076c34d0085a166c59c11d2b8c087b876d008f8ef34df453
Sep 10 16:01:42.238565 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:42.238528    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7592-aba28b" event={"ID":"6e1ce14f-bfda-4b33-9aff-fecc89a2f42a","Type":"ContainerStarted","Data":"2687a4284a55deaa8093c84bbae24fb9b5d272908be6b3c884987b2d9bab2699"}
Sep 10 16:01:42.238565 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:42.238571    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7592-aba28b" event={"ID":"6e1ce14f-bfda-4b33-9aff-fecc89a2f42a","Type":"ContainerStarted","Data":"990847e614f5050c076c34d0085a166c59c11d2b8c087b876d008f8ef34df453"}
Sep 10 16:01:42.254466 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:42.254406    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7592-aba28b" podStartSLOduration=2.254385621 podStartE2EDuration="2.254385621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:42.251764394 +0000 UTC m=+1174.924174479" watchObservedRunningTime="2026-09-10 16:01:42.254385621 +0000 UTC m=+1174.926795705"
Sep 10 16:01:42.877026 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:42.876987    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7592-aba28b"]
Sep 10 16:01:42.880726 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:42.880688    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7592-aba28b"]
Sep 10 16:01:42.991683 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:42.991593    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7592-53ee77"]
Sep 10 16:01:42.992016 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:42.992003    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e1ce14f-bfda-4b33-9aff-fecc89a2f42a" containerName="test-curl-7592-aba28b"
Sep 10 16:01:42.992064 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:42.992018    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e1ce14f-bfda-4b33-9aff-fecc89a2f42a" containerName="test-curl-7592-aba28b"
Sep 10 16:01:42.992100 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:42.992070    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e1ce14f-bfda-4b33-9aff-fecc89a2f42a" containerName="test-curl-7592-aba28b"
Sep 10 16:01:42.995167 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:42.995148    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7592-53ee77"
Sep 10 16:01:43.001360 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:43.001328    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7592-53ee77"]
Sep 10 16:01:43.139734 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:43.139689    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t8l2\" (UniqueName: \"kubernetes.io/projected/c712590b-1472-488a-b83a-a6f2978f274b-kube-api-access-6t8l2\") pod \"test-curl-7592-53ee77\" (UID: \"c712590b-1472-488a-b83a-a6f2978f274b\") " pod="opendatahub/test-curl-7592-53ee77"
Sep 10 16:01:43.240823 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:43.240787    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t8l2\" (UniqueName: \"kubernetes.io/projected/c712590b-1472-488a-b83a-a6f2978f274b-kube-api-access-6t8l2\") pod \"test-curl-7592-53ee77\" (UID: \"c712590b-1472-488a-b83a-a6f2978f274b\") " pod="opendatahub/test-curl-7592-53ee77"
Sep 10 16:01:43.244464 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:01:43.244389    2572 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7592-aba28b" container="test-curl-7592-aba28b"
Sep 10 16:01:43.249977 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:43.249944    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t8l2\" (UniqueName: \"kubernetes.io/projected/c712590b-1472-488a-b83a-a6f2978f274b-kube-api-access-6t8l2\") pod \"test-curl-7592-53ee77\" (UID: \"c712590b-1472-488a-b83a-a6f2978f274b\") " pod="opendatahub/test-curl-7592-53ee77"
Sep 10 16:01:43.306860 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:43.306818    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7592-53ee77"
Sep 10 16:01:43.579405 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:43.579373    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7592-53ee77"]
Sep 10 16:01:43.581108 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:01:43.581078    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc712590b_1472_488a_b83a_a6f2978f274b.slice/crio-d076fde88d541c763e92dd076201a88d8065085ca909b2dc3f7d7d76f889b009 WatchSource:0}: Error finding container d076fde88d541c763e92dd076201a88d8065085ca909b2dc3f7d7d76f889b009: Status 404 returned error can't find the container with id d076fde88d541c763e92dd076201a88d8065085ca909b2dc3f7d7d76f889b009
Sep 10 16:01:44.251476 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:44.251432    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7592-53ee77" event={"ID":"c712590b-1472-488a-b83a-a6f2978f274b","Type":"ContainerStarted","Data":"d42f796492227d14e9c22e97db01c6032a285669e8e1b5d0beceba64635ee625"}
Sep 10 16:01:44.251956 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:44.251466    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7592-aba28b" podUID="6e1ce14f-bfda-4b33-9aff-fecc89a2f42a" containerName="test-curl-7592-aba28b" containerID="cri-o://2687a4284a55deaa8093c84bbae24fb9b5d272908be6b3c884987b2d9bab2699" gracePeriod=2
Sep 10 16:01:44.251956 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:44.251487    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7592-53ee77" event={"ID":"c712590b-1472-488a-b83a-a6f2978f274b","Type":"ContainerStarted","Data":"d076fde88d541c763e92dd076201a88d8065085ca909b2dc3f7d7d76f889b009"}
Sep 10 16:01:44.266971 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:44.266915    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7592-53ee77" podStartSLOduration=2.26689938 podStartE2EDuration="2.26689938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:44.265040721 +0000 UTC m=+1176.937450803" watchObservedRunningTime="2026-09-10 16:01:44.26689938 +0000 UTC m=+1176.939309462"
Sep 10 16:01:44.909313 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:44.909270    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7592-53ee77"]
Sep 10 16:01:44.913385 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:44.913354    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7592-53ee77"]
Sep 10 16:01:45.035553 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:45.035457    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7592-1ab99c"]
Sep 10 16:01:45.035832 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:45.035819    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c712590b-1472-488a-b83a-a6f2978f274b" containerName="test-curl-7592-53ee77"
Sep 10 16:01:45.035880 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:45.035834    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c712590b-1472-488a-b83a-a6f2978f274b" containerName="test-curl-7592-53ee77"
Sep 10 16:01:45.035919 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:45.035911    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c712590b-1472-488a-b83a-a6f2978f274b" containerName="test-curl-7592-53ee77"
Sep 10 16:01:45.038990 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:45.038972    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7592-1ab99c"
Sep 10 16:01:45.044861 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:45.044826    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7592-1ab99c"]
Sep 10 16:01:45.160537 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:45.160495    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmx2t\" (UniqueName: \"kubernetes.io/projected/bca1c347-bce4-4115-96d3-a5bb1e44bb4e-kube-api-access-gmx2t\") pod \"test-curl-7592-1ab99c\" (UID: \"bca1c347-bce4-4115-96d3-a5bb1e44bb4e\") " pod="opendatahub/test-curl-7592-1ab99c"
Sep 10 16:01:45.257428 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:01:45.257397    2572 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7592-53ee77" container="test-curl-7592-53ee77"
Sep 10 16:01:45.261326 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:45.261284    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmx2t\" (UniqueName: \"kubernetes.io/projected/bca1c347-bce4-4115-96d3-a5bb1e44bb4e-kube-api-access-gmx2t\") pod \"test-curl-7592-1ab99c\" (UID: \"bca1c347-bce4-4115-96d3-a5bb1e44bb4e\") " pod="opendatahub/test-curl-7592-1ab99c"
Sep 10 16:01:45.270229 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:45.270203    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmx2t\" (UniqueName: \"kubernetes.io/projected/bca1c347-bce4-4115-96d3-a5bb1e44bb4e-kube-api-access-gmx2t\") pod \"test-curl-7592-1ab99c\" (UID: \"bca1c347-bce4-4115-96d3-a5bb1e44bb4e\") " pod="opendatahub/test-curl-7592-1ab99c"
Sep 10 16:01:45.350478 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:45.350435    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7592-1ab99c"
Sep 10 16:01:45.622462 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:45.622365    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7592-1ab99c"]
Sep 10 16:01:45.626528 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:01:45.626496    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbca1c347_bce4_4115_96d3_a5bb1e44bb4e.slice/crio-979063f414cb78905a72cf91f9abaf70f1921d1c308d772b3d465c1ec3f9a1d1 WatchSource:0}: Error finding container 979063f414cb78905a72cf91f9abaf70f1921d1c308d772b3d465c1ec3f9a1d1: Status 404 returned error can't find the container with id 979063f414cb78905a72cf91f9abaf70f1921d1c308d772b3d465c1ec3f9a1d1
Sep 10 16:01:46.264687 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:46.264652    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7592-1ab99c" event={"ID":"bca1c347-bce4-4115-96d3-a5bb1e44bb4e","Type":"ContainerStarted","Data":"5f5d731d461e4865cc52d3974fc59027c268682f7626a711fdfd79028895976c"}
Sep 10 16:01:46.265025 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:46.264701    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7592-1ab99c" event={"ID":"bca1c347-bce4-4115-96d3-a5bb1e44bb4e","Type":"ContainerStarted","Data":"979063f414cb78905a72cf91f9abaf70f1921d1c308d772b3d465c1ec3f9a1d1"}
Sep 10 16:01:46.265025 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:46.264790    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7592-53ee77" podUID="c712590b-1472-488a-b83a-a6f2978f274b" containerName="test-curl-7592-53ee77" containerID="cri-o://d42f796492227d14e9c22e97db01c6032a285669e8e1b5d0beceba64635ee625" gracePeriod=2
Sep 10 16:01:46.278982 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:46.278935    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7592-1ab99c" podStartSLOduration=1.278918835 podStartE2EDuration="1.278918835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:46.277785721 +0000 UTC m=+1178.950195804" watchObservedRunningTime="2026-09-10 16:01:46.278918835 +0000 UTC m=+1178.951328942"
Sep 10 16:01:46.439612 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:46.439585    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7592-aba28b"
Sep 10 16:01:46.575827 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:46.575786    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9kjq\" (UniqueName: \"kubernetes.io/projected/6e1ce14f-bfda-4b33-9aff-fecc89a2f42a-kube-api-access-h9kjq\") pod \"6e1ce14f-bfda-4b33-9aff-fecc89a2f42a\" (UID: \"6e1ce14f-bfda-4b33-9aff-fecc89a2f42a\") "
Sep 10 16:01:46.577911 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:46.577885    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1ce14f-bfda-4b33-9aff-fecc89a2f42a-kube-api-access-h9kjq" (OuterVolumeSpecName: "kube-api-access-h9kjq") pod "6e1ce14f-bfda-4b33-9aff-fecc89a2f42a" (UID: "6e1ce14f-bfda-4b33-9aff-fecc89a2f42a"). InnerVolumeSpecName "kube-api-access-h9kjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:01:46.676586 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:46.676554    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9kjq\" (UniqueName: \"kubernetes.io/projected/6e1ce14f-bfda-4b33-9aff-fecc89a2f42a-kube-api-access-h9kjq\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:01:46.899060 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:46.898935    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7592-1ab99c"]
Sep 10 16:01:46.902333 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:46.902282    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7592-1ab99c"]
Sep 10 16:01:47.023337 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:47.023278    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7592-8221bc"]
Sep 10 16:01:47.023695 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:47.023681    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca1c347-bce4-4115-96d3-a5bb1e44bb4e" containerName="test-curl-7592-1ab99c"
Sep 10 16:01:47.023744 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:47.023698    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca1c347-bce4-4115-96d3-a5bb1e44bb4e" containerName="test-curl-7592-1ab99c"
Sep 10 16:01:47.023795 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:47.023785    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bca1c347-bce4-4115-96d3-a5bb1e44bb4e" containerName="test-curl-7592-1ab99c"
Sep 10 16:01:47.026754 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:47.026736    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7592-8221bc"
Sep 10 16:01:47.031271 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:47.031243    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7592-8221bc"]
Sep 10 16:01:47.181840 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:47.181723    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-595qs\" (UniqueName: \"kubernetes.io/projected/2a3758c2-c0dd-4f8b-a552-4e35b63470b6-kube-api-access-595qs\") pod \"test-curl-7592-8221bc\" (UID: \"2a3758c2-c0dd-4f8b-a552-4e35b63470b6\") " pod="opendatahub/test-curl-7592-8221bc"
Sep 10 16:01:47.271858 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:47.271821    2572 generic.go:356] "Generic (PLEG): container finished" podID="6e1ce14f-bfda-4b33-9aff-fecc89a2f42a" containerID="2687a4284a55deaa8093c84bbae24fb9b5d272908be6b3c884987b2d9bab2699" exitCode=137
Sep 10 16:01:47.272357 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:47.271863    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7592-aba28b"
Sep 10 16:01:47.272357 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:47.271913    2572 scope.go:117] "RemoveContainer" containerID="2687a4284a55deaa8093c84bbae24fb9b5d272908be6b3c884987b2d9bab2699"
Sep 10 16:01:47.272357 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:01:47.272257    2572 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7592-1ab99c" container="test-curl-7592-1ab99c"
Sep 10 16:01:47.281415 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:47.281397    2572 scope.go:117] "RemoveContainer" containerID="2687a4284a55deaa8093c84bbae24fb9b5d272908be6b3c884987b2d9bab2699"
Sep 10 16:01:47.281733 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:01:47.281714    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2687a4284a55deaa8093c84bbae24fb9b5d272908be6b3c884987b2d9bab2699\": container with ID starting with 2687a4284a55deaa8093c84bbae24fb9b5d272908be6b3c884987b2d9bab2699 not found: ID does not exist" containerID="2687a4284a55deaa8093c84bbae24fb9b5d272908be6b3c884987b2d9bab2699"
Sep 10 16:01:47.281798 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:47.281746    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2687a4284a55deaa8093c84bbae24fb9b5d272908be6b3c884987b2d9bab2699"} err="failed to get container status \"2687a4284a55deaa8093c84bbae24fb9b5d272908be6b3c884987b2d9bab2699\": rpc error: code = NotFound desc = could not find container \"2687a4284a55deaa8093c84bbae24fb9b5d272908be6b3c884987b2d9bab2699\": container with ID starting with 2687a4284a55deaa8093c84bbae24fb9b5d272908be6b3c884987b2d9bab2699 not found: ID does not exist"
Sep 10 16:01:47.281930 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:01:47.281912    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7592-aba28b\" is forbidden: User \"system:node:ip-10-0-134-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-36.ec2.internal' and this object" podUID="6e1ce14f-bfda-4b33-9aff-fecc89a2f42a" pod="opendatahub/test-curl-7592-aba28b"
Sep 10 16:01:47.282367 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:47.282347    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-595qs\" (UniqueName: \"kubernetes.io/projected/2a3758c2-c0dd-4f8b-a552-4e35b63470b6-kube-api-access-595qs\") pod \"test-curl-7592-8221bc\" (UID: \"2a3758c2-c0dd-4f8b-a552-4e35b63470b6\") " pod="opendatahub/test-curl-7592-8221bc"
Sep 10 16:01:47.291824 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:47.291803    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-595qs\" (UniqueName: \"kubernetes.io/projected/2a3758c2-c0dd-4f8b-a552-4e35b63470b6-kube-api-access-595qs\") pod \"test-curl-7592-8221bc\" (UID: \"2a3758c2-c0dd-4f8b-a552-4e35b63470b6\") " pod="opendatahub/test-curl-7592-8221bc"
Sep 10 16:01:47.337727 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:47.337687    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7592-8221bc"
Sep 10 16:01:47.613428 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:47.613397    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7592-8221bc"]
Sep 10 16:01:47.615127 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:01:47.615096    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a3758c2_c0dd_4f8b_a552_4e35b63470b6.slice/crio-bec693d40b4a6fe418c2a304a62b9c758eceb0dc94dd35ec51940782a08c5269 WatchSource:0}: Error finding container bec693d40b4a6fe418c2a304a62b9c758eceb0dc94dd35ec51940782a08c5269: Status 404 returned error can't find the container with id bec693d40b4a6fe418c2a304a62b9c758eceb0dc94dd35ec51940782a08c5269
Sep 10 16:01:47.991986 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:01:47.991890    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7592-aba28b\" is forbidden: User \"system:node:ip-10-0-134-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-36.ec2.internal' and this object" podUID="6e1ce14f-bfda-4b33-9aff-fecc89a2f42a" pod="opendatahub/test-curl-7592-aba28b"
Sep 10 16:01:47.992774 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:47.992751    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e1ce14f-bfda-4b33-9aff-fecc89a2f42a" path="/var/lib/kubelet/pods/6e1ce14f-bfda-4b33-9aff-fecc89a2f42a/volumes"
Sep 10 16:01:48.281423 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:48.281393    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7592-8221bc" event={"ID":"2a3758c2-c0dd-4f8b-a552-4e35b63470b6","Type":"ContainerStarted","Data":"05a50559318d6f79f8518ccf09fb3a0ce48b3db1b002c22ae1d70cae695ae8dc"}
Sep 10 16:01:48.281773 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:48.281438    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7592-8221bc" event={"ID":"2a3758c2-c0dd-4f8b-a552-4e35b63470b6","Type":"ContainerStarted","Data":"bec693d40b4a6fe418c2a304a62b9c758eceb0dc94dd35ec51940782a08c5269"}
Sep 10 16:01:48.281773 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:48.281441    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7592-1ab99c" podUID="bca1c347-bce4-4115-96d3-a5bb1e44bb4e" containerName="test-curl-7592-1ab99c" containerID="cri-o://5f5d731d461e4865cc52d3974fc59027c268682f7626a711fdfd79028895976c" gracePeriod=2
Sep 10 16:01:48.303577 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:48.303519    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7592-8221bc" podStartSLOduration=1.30350079 podStartE2EDuration="1.30350079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:48.300992087 +0000 UTC m=+1180.973402169" watchObservedRunningTime="2026-09-10 16:01:48.30350079 +0000 UTC m=+1180.975910874"
Sep 10 16:01:48.434169 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:48.434142    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7592-53ee77"
Sep 10 16:01:48.593409 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:48.593374    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t8l2\" (UniqueName: \"kubernetes.io/projected/c712590b-1472-488a-b83a-a6f2978f274b-kube-api-access-6t8l2\") pod \"c712590b-1472-488a-b83a-a6f2978f274b\" (UID: \"c712590b-1472-488a-b83a-a6f2978f274b\") "
Sep 10 16:01:48.595556 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:48.595512    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c712590b-1472-488a-b83a-a6f2978f274b-kube-api-access-6t8l2" (OuterVolumeSpecName: "kube-api-access-6t8l2") pod "c712590b-1472-488a-b83a-a6f2978f274b" (UID: "c712590b-1472-488a-b83a-a6f2978f274b"). InnerVolumeSpecName "kube-api-access-6t8l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:01:48.695052 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:48.695017    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t8l2\" (UniqueName: \"kubernetes.io/projected/c712590b-1472-488a-b83a-a6f2978f274b-kube-api-access-6t8l2\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:01:48.916606 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:48.916507    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7592-8221bc"]
Sep 10 16:01:48.919930 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:48.919897    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7592-8221bc"]
Sep 10 16:01:49.288201 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.288106    2572 generic.go:356] "Generic (PLEG): container finished" podID="c712590b-1472-488a-b83a-a6f2978f274b" containerID="d42f796492227d14e9c22e97db01c6032a285669e8e1b5d0beceba64635ee625" exitCode=137
Sep 10 16:01:49.288201 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.288163    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7592-53ee77"
Sep 10 16:01:49.288725 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.288209    2572 scope.go:117] "RemoveContainer" containerID="d42f796492227d14e9c22e97db01c6032a285669e8e1b5d0beceba64635ee625"
Sep 10 16:01:49.288725 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:01:49.288526    2572 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7592-8221bc" container="test-curl-7592-8221bc"
Sep 10 16:01:49.300813 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.298748    2572 scope.go:117] "RemoveContainer" containerID="d42f796492227d14e9c22e97db01c6032a285669e8e1b5d0beceba64635ee625"
Sep 10 16:01:49.300813 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:01:49.299259    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7592-53ee77\" is forbidden: User \"system:node:ip-10-0-134-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-36.ec2.internal' and this object" podUID="c712590b-1472-488a-b83a-a6f2978f274b" pod="opendatahub/test-curl-7592-53ee77"
Sep 10 16:01:49.301255 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:01:49.301232    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42f796492227d14e9c22e97db01c6032a285669e8e1b5d0beceba64635ee625\": container with ID starting with d42f796492227d14e9c22e97db01c6032a285669e8e1b5d0beceba64635ee625 not found: ID does not exist" containerID="d42f796492227d14e9c22e97db01c6032a285669e8e1b5d0beceba64635ee625"
Sep 10 16:01:49.301330 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.301266    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42f796492227d14e9c22e97db01c6032a285669e8e1b5d0beceba64635ee625"} err="failed to get container status \"d42f796492227d14e9c22e97db01c6032a285669e8e1b5d0beceba64635ee625\": rpc error: code = NotFound desc = could not find container \"d42f796492227d14e9c22e97db01c6032a285669e8e1b5d0beceba64635ee625\": container with ID starting with d42f796492227d14e9c22e97db01c6032a285669e8e1b5d0beceba64635ee625 not found: ID does not exist"
Sep 10 16:01:49.868272 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.868237    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"]
Sep 10 16:01:49.868790 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.868767    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a3758c2-c0dd-4f8b-a552-4e35b63470b6" containerName="test-curl-7592-8221bc"
Sep 10 16:01:49.868790 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.868791    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a3758c2-c0dd-4f8b-a552-4e35b63470b6" containerName="test-curl-7592-8221bc"
Sep 10 16:01:49.868967 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.868893    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a3758c2-c0dd-4f8b-a552-4e35b63470b6" containerName="test-curl-7592-8221bc"
Sep 10 16:01:49.873246 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.873221    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:49.875418 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:01:49.875389    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7592-53ee77\" is forbidden: User \"system:node:ip-10-0-134-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-36.ec2.internal' and this object" podUID="c712590b-1472-488a-b83a-a6f2978f274b" pod="opendatahub/test-curl-7592-53ee77"
Sep 10 16:01:49.876083 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.876062    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0e6e0dcc-openshift-default-dockercfg-vq5xx\""
Sep 10 16:01:49.884929 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.884898    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"]
Sep 10 16:01:49.903373 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.903332    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-envoy\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:49.903373 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.903373    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-workload-socket\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:49.903602 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.903415    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-workload-certs\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:49.903602 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.903436    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-podinfo\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:49.903602 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.903489    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-credential-socket\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:49.903602 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.903525    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istiod-ca-cert\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:49.903602 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.903545    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kfpp\" (UniqueName: \"kubernetes.io/projected/b475aee9-ec85-45a1-8167-5b83ff12bfdf-kube-api-access-7kfpp\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:49.903778 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.903614    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-data\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:49.903778 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.903650    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-token\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:49.993546 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:49.993505    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c712590b-1472-488a-b83a-a6f2978f274b" path="/var/lib/kubelet/pods/c712590b-1472-488a-b83a-a6f2978f274b/volumes"
Sep 10 16:01:50.004730 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.004695    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istiod-ca-cert\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:50.004730 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.004737    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kfpp\" (UniqueName: \"kubernetes.io/projected/b475aee9-ec85-45a1-8167-5b83ff12bfdf-kube-api-access-7kfpp\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:50.004990 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.004780    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-data\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:50.004990 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.004797    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-token\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:50.004990 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.004817    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-envoy\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:50.004990 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.004847    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-workload-socket\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:50.004990 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.004881    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-workload-certs\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:50.004990 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.004915    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-podinfo\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:50.004990 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.004975    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-credential-socket\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:50.005731 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.005466    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-data\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:50.005731 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.005516    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-workload-certs\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:50.005731 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.005544    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istiod-ca-cert\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:50.005731 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.005595    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-workload-socket\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:50.006759 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.006728    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-credential-socket\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:50.007873 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.007845    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-envoy\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:50.008335 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.008316    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-podinfo\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:50.013657 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.013626    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kfpp\" (UniqueName: \"kubernetes.io/projected/b475aee9-ec85-45a1-8167-5b83ff12bfdf-kube-api-access-7kfpp\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:50.013828 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.013701    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-token\") pod \"e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") " pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:50.185728 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.185645    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:50.258999 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.258961    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"]
Sep 10 16:01:50.264120 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.264092    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.266886 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.266857    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-67df2b\"/\"kube-root-ca.crt\""
Sep 10 16:01:50.267639 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.267606    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-67df2b\"/\"default-dockercfg-6bf8n\""
Sep 10 16:01:50.267639 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.267626    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-67df2b\"/\"auth-test-model-67df2b-kserve-self-signed-certs\""
Sep 10 16:01:50.267821 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.267657    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-67df2b\"/\"openshift-service-ca.crt\""
Sep 10 16:01:50.274063 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.273816    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"]
Sep 10 16:01:50.296831 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.296796    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7592-8221bc" podUID="2a3758c2-c0dd-4f8b-a552-4e35b63470b6" containerName="test-curl-7592-8221bc" containerID="cri-o://05a50559318d6f79f8518ccf09fb3a0ce48b3db1b002c22ae1d70cae695ae8dc" gracePeriod=2
Sep 10 16:01:50.307957 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.307927    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-tls-certs\") pod \"auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.308081 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.308024    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-home\") pod \"auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.308081 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.308065    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-tmp-dir\") pod \"auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.308158 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.308107    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-dshm\") pod \"auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.308158 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.308142    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9spkc\" (UniqueName: \"kubernetes.io/projected/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-kube-api-access-9spkc\") pod \"auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.308253 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.308226    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-model-cache\") pod \"auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.409028 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.408995    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-model-cache\") pod \"auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.409028 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.409042    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-tls-certs\") pod \"auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.409288 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.409092    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-home\") pod \"auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.409288 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.409124    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-tmp-dir\") pod \"auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.409288 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.409165    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-dshm\") pod \"auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.409288 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.409208    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9spkc\" (UniqueName: \"kubernetes.io/projected/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-kube-api-access-9spkc\") pod \"auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.409541 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.409491    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-model-cache\") pod \"auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.409541 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.409506    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-home\") pod \"auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.409653 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.409636    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-tmp-dir\") pod \"auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.411531 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.411503    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-dshm\") pod \"auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.411653 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.411613    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-tls-certs\") pod \"auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.421274 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.421238    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9spkc\" (UniqueName: \"kubernetes.io/projected/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-kube-api-access-9spkc\") pod \"auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.443792 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.443722    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7592-1ab99c"
Sep 10 16:01:50.465378 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.465345    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"]
Sep 10 16:01:50.466943 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:01:50.466914    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb475aee9_ec85_45a1_8167_5b83ff12bfdf.slice/crio-f1540c0969d04f8a39122054cdcb733e52205d9cdada58d7c0dba00cd1d08bd4 WatchSource:0}: Error finding container f1540c0969d04f8a39122054cdcb733e52205d9cdada58d7c0dba00cd1d08bd4: Status 404 returned error can't find the container with id f1540c0969d04f8a39122054cdcb733e52205d9cdada58d7c0dba00cd1d08bd4
Sep 10 16:01:50.469350 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.469312    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:01:50.469418 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.469398    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:01:50.469456 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.469436    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:01:50.509677 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.509648    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmx2t\" (UniqueName: \"kubernetes.io/projected/bca1c347-bce4-4115-96d3-a5bb1e44bb4e-kube-api-access-gmx2t\") pod \"bca1c347-bce4-4115-96d3-a5bb1e44bb4e\" (UID: \"bca1c347-bce4-4115-96d3-a5bb1e44bb4e\") "
Sep 10 16:01:50.511782 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.511754    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca1c347-bce4-4115-96d3-a5bb1e44bb4e-kube-api-access-gmx2t" (OuterVolumeSpecName: "kube-api-access-gmx2t") pod "bca1c347-bce4-4115-96d3-a5bb1e44bb4e" (UID: "bca1c347-bce4-4115-96d3-a5bb1e44bb4e"). InnerVolumeSpecName "kube-api-access-gmx2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:01:50.577640 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.577602    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:01:50.611408 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.611365    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmx2t\" (UniqueName: \"kubernetes.io/projected/bca1c347-bce4-4115-96d3-a5bb1e44bb4e-kube-api-access-gmx2t\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:01:50.867738 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:50.867702    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"]
Sep 10 16:01:50.869244 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:01:50.869211    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57cc7f1d_14a7_4f79_81d3_bf6512daa0e7.slice/crio-3848b1fd00ca52355fe7e1b58f3137d6e97fefa89f91f67eb71385809a191a93 WatchSource:0}: Error finding container 3848b1fd00ca52355fe7e1b58f3137d6e97fefa89f91f67eb71385809a191a93: Status 404 returned error can't find the container with id 3848b1fd00ca52355fe7e1b58f3137d6e97fefa89f91f67eb71385809a191a93
Sep 10 16:01:51.303641 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:51.303600    2572 generic.go:356] "Generic (PLEG): container finished" podID="bca1c347-bce4-4115-96d3-a5bb1e44bb4e" containerID="5f5d731d461e4865cc52d3974fc59027c268682f7626a711fdfd79028895976c" exitCode=137
Sep 10 16:01:51.304103 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:51.303655    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7592-1ab99c"
Sep 10 16:01:51.304103 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:51.303703    2572 scope.go:117] "RemoveContainer" containerID="5f5d731d461e4865cc52d3974fc59027c268682f7626a711fdfd79028895976c"
Sep 10 16:01:51.305634 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:51.305605    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2" event={"ID":"b475aee9-ec85-45a1-8167-5b83ff12bfdf","Type":"ContainerStarted","Data":"c46d8a3081246e383c9196fd54afb06909956f78e1db0c5dd2be88a0dc2820dc"}
Sep 10 16:01:51.305774 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:51.305701    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2" event={"ID":"b475aee9-ec85-45a1-8167-5b83ff12bfdf","Type":"ContainerStarted","Data":"f1540c0969d04f8a39122054cdcb733e52205d9cdada58d7c0dba00cd1d08bd4"}
Sep 10 16:01:51.307361 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:51.307337    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn" event={"ID":"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7","Type":"ContainerStarted","Data":"78dffccb783190380c7146d3652a68bde421f75ab0abe067e2262a51b46cd8e2"}
Sep 10 16:01:51.307468 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:51.307374    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn" event={"ID":"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7","Type":"ContainerStarted","Data":"3848b1fd00ca52355fe7e1b58f3137d6e97fefa89f91f67eb71385809a191a93"}
Sep 10 16:01:51.313474 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:51.313456    2572 scope.go:117] "RemoveContainer" containerID="5f5d731d461e4865cc52d3974fc59027c268682f7626a711fdfd79028895976c"
Sep 10 16:01:51.313803 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:01:51.313783    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f5d731d461e4865cc52d3974fc59027c268682f7626a711fdfd79028895976c\": container with ID starting with 5f5d731d461e4865cc52d3974fc59027c268682f7626a711fdfd79028895976c not found: ID does not exist" containerID="5f5d731d461e4865cc52d3974fc59027c268682f7626a711fdfd79028895976c"
Sep 10 16:01:51.313868 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:51.313813    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f5d731d461e4865cc52d3974fc59027c268682f7626a711fdfd79028895976c"} err="failed to get container status \"5f5d731d461e4865cc52d3974fc59027c268682f7626a711fdfd79028895976c\": rpc error: code = NotFound desc = could not find container \"5f5d731d461e4865cc52d3974fc59027c268682f7626a711fdfd79028895976c\": container with ID starting with 5f5d731d461e4865cc52d3974fc59027c268682f7626a711fdfd79028895976c not found: ID does not exist"
Sep 10 16:01:51.334628 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:51.334571    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2" podStartSLOduration=2.334553153 podStartE2EDuration="2.334553153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:51.332850927 +0000 UTC m=+1184.005261012" watchObservedRunningTime="2026-09-10 16:01:51.334553153 +0000 UTC m=+1184.006963236"
Sep 10 16:01:51.335003 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:01:51.334974    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7592-1ab99c\" is forbidden: User \"system:node:ip-10-0-134-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-36.ec2.internal' and this object" podUID="bca1c347-bce4-4115-96d3-a5bb1e44bb4e" pod="opendatahub/test-curl-7592-1ab99c"
Sep 10 16:01:51.354795 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:51.354738    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn" podStartSLOduration=1.354718549 podStartE2EDuration="1.354718549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:51.353092082 +0000 UTC m=+1184.025502179" watchObservedRunningTime="2026-09-10 16:01:51.354718549 +0000 UTC m=+1184.027128736"
Sep 10 16:01:51.992963 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:51.992923    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bca1c347-bce4-4115-96d3-a5bb1e44bb4e" path="/var/lib/kubelet/pods/bca1c347-bce4-4115-96d3-a5bb1e44bb4e/volumes"
Sep 10 16:01:52.186396 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:52.186350    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:52.192175 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:52.192132    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:52.315853 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:52.315824    2572 generic.go:356] "Generic (PLEG): container finished" podID="2a3758c2-c0dd-4f8b-a552-4e35b63470b6" containerID="05a50559318d6f79f8518ccf09fb3a0ce48b3db1b002c22ae1d70cae695ae8dc" exitCode=137
Sep 10 16:01:52.323518 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:52.323491    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:52.324657 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:52.324634    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:01:52.451542 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:52.451515    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7592-8221bc"
Sep 10 16:01:52.530099 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:52.530056    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-595qs\" (UniqueName: \"kubernetes.io/projected/2a3758c2-c0dd-4f8b-a552-4e35b63470b6-kube-api-access-595qs\") pod \"2a3758c2-c0dd-4f8b-a552-4e35b63470b6\" (UID: \"2a3758c2-c0dd-4f8b-a552-4e35b63470b6\") "
Sep 10 16:01:52.532166 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:52.532131    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3758c2-c0dd-4f8b-a552-4e35b63470b6-kube-api-access-595qs" (OuterVolumeSpecName: "kube-api-access-595qs") pod "2a3758c2-c0dd-4f8b-a552-4e35b63470b6" (UID: "2a3758c2-c0dd-4f8b-a552-4e35b63470b6"). InnerVolumeSpecName "kube-api-access-595qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:01:52.631846 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:52.631748    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-595qs\" (UniqueName: \"kubernetes.io/projected/2a3758c2-c0dd-4f8b-a552-4e35b63470b6-kube-api-access-595qs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:01:53.332889 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:53.332861    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7592-8221bc"
Sep 10 16:01:53.333354 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:53.332922    2572 scope.go:117] "RemoveContainer" containerID="05a50559318d6f79f8518ccf09fb3a0ce48b3db1b002c22ae1d70cae695ae8dc"
Sep 10 16:01:53.344406 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:01:53.344377    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7592-8221bc\" is forbidden: User \"system:node:ip-10-0-134-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-36.ec2.internal' and this object" podUID="2a3758c2-c0dd-4f8b-a552-4e35b63470b6" pod="opendatahub/test-curl-7592-8221bc"
Sep 10 16:01:53.992565 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:01:53.992526    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a3758c2-c0dd-4f8b-a552-4e35b63470b6" path="/var/lib/kubelet/pods/2a3758c2-c0dd-4f8b-a552-4e35b63470b6/volumes"
Sep 10 16:02:00.578272 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:00.578222    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:02:00.578797 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:00.578290    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:02:00.590855 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:00.590830    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:02:01.398169 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:01.398136    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:02:07.881085 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:07.881054    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7sm72_8ddd4dc1-1122-41e0-a54a-67fb8008d7b9/ovn-acl-logging/0.log"
Sep 10 16:02:07.887969 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:07.887942    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7sm72_8ddd4dc1-1122-41e0-a54a-67fb8008d7b9/ovn-acl-logging/0.log"
Sep 10 16:02:08.127562 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:08.127525    2572 scope.go:117] "RemoveContainer" containerID="e011f3ed85c4a5b5e9bbbc8533b47858090643990fa47543dbb6ef66d64cffc4"
Sep 10 16:02:17.492642 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:17.492606    2572 generic.go:356] "Generic (PLEG): container finished" podID="b475aee9-ec85-45a1-8167-5b83ff12bfdf" containerID="c46d8a3081246e383c9196fd54afb06909956f78e1db0c5dd2be88a0dc2820dc" exitCode=0
Sep 10 16:02:17.493187 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:17.492685    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2" event={"ID":"b475aee9-ec85-45a1-8167-5b83ff12bfdf","Type":"ContainerDied","Data":"c46d8a3081246e383c9196fd54afb06909956f78e1db0c5dd2be88a0dc2820dc"}
Sep 10 16:02:17.493258 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:17.493228    2572 scope.go:117] "RemoveContainer" containerID="c46d8a3081246e383c9196fd54afb06909956f78e1db0c5dd2be88a0dc2820dc"
Sep 10 16:02:17.494532 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:17.494505    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:02:17.494619 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:17.494571    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:02:17.494619 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:17.494604    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:02:18.501633 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:18.501596    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2" event={"ID":"b475aee9-ec85-45a1-8167-5b83ff12bfdf","Type":"ContainerStarted","Data":"2fa50ee3673bd7329d42142802b7184cea259e991b099f296825173db1c48403"}
Sep 10 16:02:19.186337 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:19.186274    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:02:19.187578 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:19.187549    2572 patch_prober.go:28] interesting pod/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 16:02:19.187688 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:19.187602    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2" podUID="b475aee9-ec85-45a1-8167-5b83ff12bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 16:02:20.186946 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:20.186903    2572 patch_prober.go:28] interesting pod/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 16:02:20.187368 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:20.186967    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2" podUID="b475aee9-ec85-45a1-8167-5b83ff12bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 16:02:21.186213 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:21.186169    2572 patch_prober.go:28] interesting pod/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 16:02:21.186514 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:21.186236    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2" podUID="b475aee9-ec85-45a1-8167-5b83ff12bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 16:02:22.186160 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:22.186121    2572 patch_prober.go:28] interesting pod/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 16:02:22.186619 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:22.186193    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2" podUID="b475aee9-ec85-45a1-8167-5b83ff12bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 16:02:22.324250 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:22.323403    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:02:23.186277 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:23.186236    2572 patch_prober.go:28] interesting pod/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 16:02:23.186700 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:23.186357    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2" podUID="b475aee9-ec85-45a1-8167-5b83ff12bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 16:02:24.190699 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:24.190670    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:02:24.191722 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:24.191702    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:02:34.560961 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.560920    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"]
Sep 10 16:02:34.565631 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.565611    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:34.574034 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.573989    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0e6e0dcc\"/\"e2e-lifecycle-model-0e6e0dcc-kserve-self-signed-certs\""
Sep 10 16:02:34.574212 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.574068    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0e6e0dcc\"/\"openshift-service-ca.crt\""
Sep 10 16:02:34.574212 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.574095    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0e6e0dcc\"/\"default-dockercfg-zmhpc\""
Sep 10 16:02:34.574212 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.574105    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0e6e0dcc\"/\"kube-root-ca.crt\""
Sep 10 16:02:34.576064 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.576038    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"]
Sep 10 16:02:34.683131 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.683088    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc0af41-c170-4cca-806f-e849ecc86ee0-tls-certs\") pod \"e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") " pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:34.683351 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.683146    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-tmp-dir\") pod \"e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") " pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:34.683351 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.683235    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-model-cache\") pod \"e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") " pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:34.683351 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.683283    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-home\") pod \"e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") " pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:34.683351 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.683316    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-dshm\") pod \"e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") " pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:34.683351 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.683338    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89gwv\" (UniqueName: \"kubernetes.io/projected/edc0af41-c170-4cca-806f-e849ecc86ee0-kube-api-access-89gwv\") pod \"e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") " pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:34.784291 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.784246    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc0af41-c170-4cca-806f-e849ecc86ee0-tls-certs\") pod \"e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") " pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:34.784554 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.784345    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-tmp-dir\") pod \"e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") " pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:34.784554 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.784388    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-model-cache\") pod \"e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") " pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:34.784554 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.784415    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-home\") pod \"e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") " pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:34.784554 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.784436    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-dshm\") pod \"e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") " pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:34.784554 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.784464    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89gwv\" (UniqueName: \"kubernetes.io/projected/edc0af41-c170-4cca-806f-e849ecc86ee0-kube-api-access-89gwv\") pod \"e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") " pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:34.784907 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.784877    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-tmp-dir\") pod \"e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") " pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:34.784987 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.784933    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-model-cache\") pod \"e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") " pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:34.784987 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.784958    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-home\") pod \"e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") " pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:34.786898 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.786871    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-dshm\") pod \"e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") " pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:34.787051 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.787033    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc0af41-c170-4cca-806f-e849ecc86ee0-tls-certs\") pod \"e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") " pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:34.793442 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.793406    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89gwv\" (UniqueName: \"kubernetes.io/projected/edc0af41-c170-4cca-806f-e849ecc86ee0-kube-api-access-89gwv\") pod \"e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") " pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:34.876543 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:34.876445    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:35.156997 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:35.156960    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"]
Sep 10 16:02:35.158127 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:02:35.158099    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedc0af41_c170_4cca_806f_e849ecc86ee0.slice/crio-ee2c008110342ba5f6502596f77aeea6b62e9729dbdf434a0643bcf028d84fe1 WatchSource:0}: Error finding container ee2c008110342ba5f6502596f77aeea6b62e9729dbdf434a0643bcf028d84fe1: Status 404 returned error can't find the container with id ee2c008110342ba5f6502596f77aeea6b62e9729dbdf434a0643bcf028d84fe1
Sep 10 16:02:35.618810 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:35.618770    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt" event={"ID":"edc0af41-c170-4cca-806f-e849ecc86ee0","Type":"ContainerStarted","Data":"68194e1b8611512f1d9f0c1a1f9add9c890fffa961a0c035b5c144698872e5c8"}
Sep 10 16:02:35.618810 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:35.618817    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt" event={"ID":"edc0af41-c170-4cca-806f-e849ecc86ee0","Type":"ContainerStarted","Data":"ee2c008110342ba5f6502596f77aeea6b62e9729dbdf434a0643bcf028d84fe1"}
Sep 10 16:02:35.640062 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:35.639994    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt" podStartSLOduration=1.6399746469999998 podStartE2EDuration="1.639974647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:35.638255021 +0000 UTC m=+1228.310665106" watchObservedRunningTime="2026-09-10 16:02:35.639974647 +0000 UTC m=+1228.312384730"
Sep 10 16:02:44.876837 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:44.876798    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:44.877262 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:44.876859    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:44.889498 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:44.889467    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:02:45.690222 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:02:45.690193    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:03:42.421868 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:42.421832    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"]
Sep 10 16:03:53.470447 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:53.470406    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7595-a8761f"]
Sep 10 16:03:53.473978 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:53.473956    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7595-a8761f"
Sep 10 16:03:53.476355 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:53.476326    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-j2pv4\""
Sep 10 16:03:53.480882 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:53.480855    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7595-a8761f"]
Sep 10 16:03:53.498384 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:53.498344    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26j7v\" (UniqueName: \"kubernetes.io/projected/5dea626c-f27e-4da4-8677-f2a94cf342b3-kube-api-access-26j7v\") pod \"test-curl-7595-a8761f\" (UID: \"5dea626c-f27e-4da4-8677-f2a94cf342b3\") " pod="opendatahub/test-curl-7595-a8761f"
Sep 10 16:03:53.599596 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:53.599550    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26j7v\" (UniqueName: \"kubernetes.io/projected/5dea626c-f27e-4da4-8677-f2a94cf342b3-kube-api-access-26j7v\") pod \"test-curl-7595-a8761f\" (UID: \"5dea626c-f27e-4da4-8677-f2a94cf342b3\") " pod="opendatahub/test-curl-7595-a8761f"
Sep 10 16:03:53.608007 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:53.607972    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26j7v\" (UniqueName: \"kubernetes.io/projected/5dea626c-f27e-4da4-8677-f2a94cf342b3-kube-api-access-26j7v\") pod \"test-curl-7595-a8761f\" (UID: \"5dea626c-f27e-4da4-8677-f2a94cf342b3\") " pod="opendatahub/test-curl-7595-a8761f"
Sep 10 16:03:53.784684 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:53.784581    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7595-a8761f"
Sep 10 16:03:54.057575 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:54.057544    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7595-a8761f"]
Sep 10 16:03:54.058991 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:03:54.058965    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5dea626c_f27e_4da4_8677_f2a94cf342b3.slice/crio-c73e4e2f1bae16bc44846c284900efe54ab3683567beb3787376e93d2247717d WatchSource:0}: Error finding container c73e4e2f1bae16bc44846c284900efe54ab3683567beb3787376e93d2247717d: Status 404 returned error can't find the container with id c73e4e2f1bae16bc44846c284900efe54ab3683567beb3787376e93d2247717d
Sep 10 16:03:54.099748 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:54.099716    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7595-a8761f" event={"ID":"5dea626c-f27e-4da4-8677-f2a94cf342b3","Type":"ContainerStarted","Data":"c73e4e2f1bae16bc44846c284900efe54ab3683567beb3787376e93d2247717d"}
Sep 10 16:03:55.108556 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:55.108516    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7595-a8761f" event={"ID":"5dea626c-f27e-4da4-8677-f2a94cf342b3","Type":"ContainerStarted","Data":"27446002d5c4339e9fe3f216d908cc18d5df3fb46402a697abe55f9089179477"}
Sep 10 16:03:55.126433 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:55.126373    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7595-a8761f" podStartSLOduration=2.1263540069999998 podStartE2EDuration="2.126354007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:03:55.124830766 +0000 UTC m=+1307.797240849" watchObservedRunningTime="2026-09-10 16:03:55.126354007 +0000 UTC m=+1307.798764089"
Sep 10 16:03:55.754761 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:55.754724    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7595-a8761f"]
Sep 10 16:03:55.761010 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:55.760980    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7595-a8761f"]
Sep 10 16:03:56.113907 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:03:56.113873    2572 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7595-a8761f" container="test-curl-7595-a8761f"
Sep 10 16:03:57.120246 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.120196    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7595-a8761f" podUID="5dea626c-f27e-4da4-8677-f2a94cf342b3" containerName="test-curl-7595-a8761f" containerID="cri-o://27446002d5c4339e9fe3f216d908cc18d5df3fb46402a697abe55f9089179477" gracePeriod=2
Sep 10 16:03:57.244739 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.244701    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"]
Sep 10 16:03:57.442970 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.442870    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt" podUID="edc0af41-c170-4cca-806f-e849ecc86ee0" containerName="main" containerID="cri-o://68194e1b8611512f1d9f0c1a1f9add9c890fffa961a0c035b5c144698872e5c8" gracePeriod=45
Sep 10 16:03:57.688367 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.688336    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:03:57.739832 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.739740    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-model-cache\") pod \"edc0af41-c170-4cca-806f-e849ecc86ee0\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") "
Sep 10 16:03:57.740001 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.739840    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-home\") pod \"edc0af41-c170-4cca-806f-e849ecc86ee0\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") "
Sep 10 16:03:57.740001 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.739877    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc0af41-c170-4cca-806f-e849ecc86ee0-tls-certs\") pod \"edc0af41-c170-4cca-806f-e849ecc86ee0\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") "
Sep 10 16:03:57.740001 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.739941    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-tmp-dir\") pod \"edc0af41-c170-4cca-806f-e849ecc86ee0\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") "
Sep 10 16:03:57.740001 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.739978    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-dshm\") pod \"edc0af41-c170-4cca-806f-e849ecc86ee0\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") "
Sep 10 16:03:57.740243 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.740007    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89gwv\" (UniqueName: \"kubernetes.io/projected/edc0af41-c170-4cca-806f-e849ecc86ee0-kube-api-access-89gwv\") pod \"edc0af41-c170-4cca-806f-e849ecc86ee0\" (UID: \"edc0af41-c170-4cca-806f-e849ecc86ee0\") "
Sep 10 16:03:57.740243 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.740021    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-model-cache" (OuterVolumeSpecName: "model-cache") pod "edc0af41-c170-4cca-806f-e849ecc86ee0" (UID: "edc0af41-c170-4cca-806f-e849ecc86ee0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:03:57.740243 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.740115    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-home" (OuterVolumeSpecName: "home") pod "edc0af41-c170-4cca-806f-e849ecc86ee0" (UID: "edc0af41-c170-4cca-806f-e849ecc86ee0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:03:57.740476 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.740346    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "edc0af41-c170-4cca-806f-e849ecc86ee0" (UID: "edc0af41-c170-4cca-806f-e849ecc86ee0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:03:57.740476 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.740455    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-tmp-dir\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:03:57.740476 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.740473    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-model-cache\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:03:57.740610 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.740489    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-home\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:03:57.742648 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.742613    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edc0af41-c170-4cca-806f-e849ecc86ee0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "edc0af41-c170-4cca-806f-e849ecc86ee0" (UID: "edc0af41-c170-4cca-806f-e849ecc86ee0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:03:57.742648 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.742614    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-dshm" (OuterVolumeSpecName: "dshm") pod "edc0af41-c170-4cca-806f-e849ecc86ee0" (UID: "edc0af41-c170-4cca-806f-e849ecc86ee0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:03:57.742841 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.742748    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc0af41-c170-4cca-806f-e849ecc86ee0-kube-api-access-89gwv" (OuterVolumeSpecName: "kube-api-access-89gwv") pod "edc0af41-c170-4cca-806f-e849ecc86ee0" (UID: "edc0af41-c170-4cca-806f-e849ecc86ee0"). InnerVolumeSpecName "kube-api-access-89gwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:03:57.841441 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.841406    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc0af41-c170-4cca-806f-e849ecc86ee0-tls-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:03:57.841441 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.841437    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc0af41-c170-4cca-806f-e849ecc86ee0-dshm\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:03:57.841441 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:57.841446    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89gwv\" (UniqueName: \"kubernetes.io/projected/edc0af41-c170-4cca-806f-e849ecc86ee0-kube-api-access-89gwv\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:03:58.127318 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:58.127263    2572 generic.go:356] "Generic (PLEG): container finished" podID="edc0af41-c170-4cca-806f-e849ecc86ee0" containerID="68194e1b8611512f1d9f0c1a1f9add9c890fffa961a0c035b5c144698872e5c8" exitCode=0
Sep 10 16:03:58.127717 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:58.127357    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"
Sep 10 16:03:58.127717 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:58.127350    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt" event={"ID":"edc0af41-c170-4cca-806f-e849ecc86ee0","Type":"ContainerDied","Data":"68194e1b8611512f1d9f0c1a1f9add9c890fffa961a0c035b5c144698872e5c8"}
Sep 10 16:03:58.127717 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:58.127429    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt" event={"ID":"edc0af41-c170-4cca-806f-e849ecc86ee0","Type":"ContainerDied","Data":"ee2c008110342ba5f6502596f77aeea6b62e9729dbdf434a0643bcf028d84fe1"}
Sep 10 16:03:58.127717 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:58.127456    2572 scope.go:117] "RemoveContainer" containerID="68194e1b8611512f1d9f0c1a1f9add9c890fffa961a0c035b5c144698872e5c8"
Sep 10 16:03:58.137094 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:58.137073    2572 scope.go:117] "RemoveContainer" containerID="68194e1b8611512f1d9f0c1a1f9add9c890fffa961a0c035b5c144698872e5c8"
Sep 10 16:03:58.137409 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:03:58.137385    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68194e1b8611512f1d9f0c1a1f9add9c890fffa961a0c035b5c144698872e5c8\": container with ID starting with 68194e1b8611512f1d9f0c1a1f9add9c890fffa961a0c035b5c144698872e5c8 not found: ID does not exist" containerID="68194e1b8611512f1d9f0c1a1f9add9c890fffa961a0c035b5c144698872e5c8"
Sep 10 16:03:58.137515 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:58.137414    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68194e1b8611512f1d9f0c1a1f9add9c890fffa961a0c035b5c144698872e5c8"} err="failed to get container status \"68194e1b8611512f1d9f0c1a1f9add9c890fffa961a0c035b5c144698872e5c8\": rpc error: code = NotFound desc = could not find container \"68194e1b8611512f1d9f0c1a1f9add9c890fffa961a0c035b5c144698872e5c8\": container with ID starting with 68194e1b8611512f1d9f0c1a1f9add9c890fffa961a0c035b5c144698872e5c8 not found: ID does not exist"
Sep 10 16:03:58.157623 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:58.157592    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"]
Sep 10 16:03:58.167557 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:58.167506    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-0e6e0dcc/e2e-lifecycle-model-0e6e0dcc-kserve-848478b56c-f9lgt"]
Sep 10 16:03:58.191761 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:58.191728    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"]
Sep 10 16:03:59.310727 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:59.310701    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7595-a8761f"
Sep 10 16:03:59.355131 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:59.355088    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26j7v\" (UniqueName: \"kubernetes.io/projected/5dea626c-f27e-4da4-8677-f2a94cf342b3-kube-api-access-26j7v\") pod \"5dea626c-f27e-4da4-8677-f2a94cf342b3\" (UID: \"5dea626c-f27e-4da4-8677-f2a94cf342b3\") "
Sep 10 16:03:59.357205 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:59.357172    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dea626c-f27e-4da4-8677-f2a94cf342b3-kube-api-access-26j7v" (OuterVolumeSpecName: "kube-api-access-26j7v") pod "5dea626c-f27e-4da4-8677-f2a94cf342b3" (UID: "5dea626c-f27e-4da4-8677-f2a94cf342b3"). InnerVolumeSpecName "kube-api-access-26j7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:03:59.456941 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:59.456852    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26j7v\" (UniqueName: \"kubernetes.io/projected/5dea626c-f27e-4da4-8677-f2a94cf342b3-kube-api-access-26j7v\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:03:59.992868 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:59.992823    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dea626c-f27e-4da4-8677-f2a94cf342b3" path="/var/lib/kubelet/pods/5dea626c-f27e-4da4-8677-f2a94cf342b3/volumes"
Sep 10 16:03:59.993375 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:03:59.993352    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edc0af41-c170-4cca-806f-e849ecc86ee0" path="/var/lib/kubelet/pods/edc0af41-c170-4cca-806f-e849ecc86ee0/volumes"
Sep 10 16:04:00.141944 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:00.141903    2572 generic.go:356] "Generic (PLEG): container finished" podID="5dea626c-f27e-4da4-8677-f2a94cf342b3" containerID="27446002d5c4339e9fe3f216d908cc18d5df3fb46402a697abe55f9089179477" exitCode=137
Sep 10 16:04:00.141944 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:00.141947    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7595-a8761f"
Sep 10 16:04:00.142188 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:00.141982    2572 scope.go:117] "RemoveContainer" containerID="27446002d5c4339e9fe3f216d908cc18d5df3fb46402a697abe55f9089179477"
Sep 10 16:04:00.151269 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:00.151249    2572 scope.go:117] "RemoveContainer" containerID="27446002d5c4339e9fe3f216d908cc18d5df3fb46402a697abe55f9089179477"
Sep 10 16:04:00.151618 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:04:00.151599    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27446002d5c4339e9fe3f216d908cc18d5df3fb46402a697abe55f9089179477\": container with ID starting with 27446002d5c4339e9fe3f216d908cc18d5df3fb46402a697abe55f9089179477 not found: ID does not exist" containerID="27446002d5c4339e9fe3f216d908cc18d5df3fb46402a697abe55f9089179477"
Sep 10 16:04:00.151670 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:00.151629    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27446002d5c4339e9fe3f216d908cc18d5df3fb46402a697abe55f9089179477"} err="failed to get container status \"27446002d5c4339e9fe3f216d908cc18d5df3fb46402a697abe55f9089179477\": rpc error: code = NotFound desc = could not find container \"27446002d5c4339e9fe3f216d908cc18d5df3fb46402a697abe55f9089179477\": container with ID starting with 27446002d5c4339e9fe3f216d908cc18d5df3fb46402a697abe55f9089179477 not found: ID does not exist"
Sep 10 16:04:08.191014 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:08.190979    2572 scope.go:117] "RemoveContainer" containerID="b5330048f62d6c686a7928fe05a30af4fdf6679e9a3618af4011bd0e97f27d27"
Sep 10 16:04:09.675234 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.675193    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"]
Sep 10 16:04:09.675789 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.675770    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dea626c-f27e-4da4-8677-f2a94cf342b3" containerName="test-curl-7595-a8761f"
Sep 10 16:04:09.675833 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.675793    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dea626c-f27e-4da4-8677-f2a94cf342b3" containerName="test-curl-7595-a8761f"
Sep 10 16:04:09.675833 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.675822    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc0af41-c170-4cca-806f-e849ecc86ee0" containerName="main"
Sep 10 16:04:09.675833 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.675830    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc0af41-c170-4cca-806f-e849ecc86ee0" containerName="main"
Sep 10 16:04:09.675933 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.675923    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="edc0af41-c170-4cca-806f-e849ecc86ee0" containerName="main"
Sep 10 16:04:09.675967 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.675937    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dea626c-f27e-4da4-8677-f2a94cf342b3" containerName="test-curl-7595-a8761f"
Sep 10 16:04:09.680898 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.680874    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:09.684291 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.684270    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-50ed73\"/\"sub-test-model-50ed73-kserve-self-signed-certs\""
Sep 10 16:04:09.691104 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.691076    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"]
Sep 10 16:04:09.762774 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.762728    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-home\") pod \"sub-test-model-50ed73-kserve-f4fd64795-bl5bm\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:09.762989 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.762788    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36689670-76ce-4696-96d8-2e238c562ab3-tls-certs\") pod \"sub-test-model-50ed73-kserve-f4fd64795-bl5bm\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:09.762989 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.762846    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-dshm\") pod \"sub-test-model-50ed73-kserve-f4fd64795-bl5bm\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:09.762989 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.762908    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz4mh\" (UniqueName: \"kubernetes.io/projected/36689670-76ce-4696-96d8-2e238c562ab3-kube-api-access-kz4mh\") pod \"sub-test-model-50ed73-kserve-f4fd64795-bl5bm\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:09.763166 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.763007    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-tmp-dir\") pod \"sub-test-model-50ed73-kserve-f4fd64795-bl5bm\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:09.763166 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.763053    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-model-cache\") pod \"sub-test-model-50ed73-kserve-f4fd64795-bl5bm\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:09.864635 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.864593    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36689670-76ce-4696-96d8-2e238c562ab3-tls-certs\") pod \"sub-test-model-50ed73-kserve-f4fd64795-bl5bm\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:09.864635 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.864635    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-dshm\") pod \"sub-test-model-50ed73-kserve-f4fd64795-bl5bm\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:09.864874 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.864658    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz4mh\" (UniqueName: \"kubernetes.io/projected/36689670-76ce-4696-96d8-2e238c562ab3-kube-api-access-kz4mh\") pod \"sub-test-model-50ed73-kserve-f4fd64795-bl5bm\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:09.864874 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.864716    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-tmp-dir\") pod \"sub-test-model-50ed73-kserve-f4fd64795-bl5bm\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:09.864874 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.864748    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-model-cache\") pod \"sub-test-model-50ed73-kserve-f4fd64795-bl5bm\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:09.864874 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.864822    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-home\") pod \"sub-test-model-50ed73-kserve-f4fd64795-bl5bm\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:09.865199 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.865174    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-model-cache\") pod \"sub-test-model-50ed73-kserve-f4fd64795-bl5bm\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:09.865280 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.865230    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-home\") pod \"sub-test-model-50ed73-kserve-f4fd64795-bl5bm\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:09.865280 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.865251    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-tmp-dir\") pod \"sub-test-model-50ed73-kserve-f4fd64795-bl5bm\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:09.866999 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.866972    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-dshm\") pod \"sub-test-model-50ed73-kserve-f4fd64795-bl5bm\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:09.867155 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.867126    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36689670-76ce-4696-96d8-2e238c562ab3-tls-certs\") pod \"sub-test-model-50ed73-kserve-f4fd64795-bl5bm\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:09.873687 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.873657    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz4mh\" (UniqueName: \"kubernetes.io/projected/36689670-76ce-4696-96d8-2e238c562ab3-kube-api-access-kz4mh\") pod \"sub-test-model-50ed73-kserve-f4fd64795-bl5bm\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:09.991636 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:09.991547    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:10.268840 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:10.268744    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"]
Sep 10 16:04:10.271997 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:04:10.271965    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36689670_76ce_4696_96d8_2e238c562ab3.slice/crio-89810ebed634192561b943a81c6a1fcb763b253429871b9104f0649dcda78a0a WatchSource:0}: Error finding container 89810ebed634192561b943a81c6a1fcb763b253429871b9104f0649dcda78a0a: Status 404 returned error can't find the container with id 89810ebed634192561b943a81c6a1fcb763b253429871b9104f0649dcda78a0a
Sep 10 16:04:11.213545 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:11.213504    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm" event={"ID":"36689670-76ce-4696-96d8-2e238c562ab3","Type":"ContainerStarted","Data":"737fd6a77a8939299c171962121429320c4d56e6218acae5dbc4d9b1e0551f95"}
Sep 10 16:04:11.213545 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:11.213552    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm" event={"ID":"36689670-76ce-4696-96d8-2e238c562ab3","Type":"ContainerStarted","Data":"89810ebed634192561b943a81c6a1fcb763b253429871b9104f0649dcda78a0a"}
Sep 10 16:04:11.242205 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:11.242143    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm" podStartSLOduration=2.242123655 podStartE2EDuration="2.242123655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:04:11.239418246 +0000 UTC m=+1323.911828330" watchObservedRunningTime="2026-09-10 16:04:11.242123655 +0000 UTC m=+1323.914533738"
Sep 10 16:04:12.264531 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.264481    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x" podUID="227622f8-939d-4ff5-a22f-4662135c4ccf" containerName="main" containerID="cri-o://1ea403e3308d2130f43804e80871ba670f723e7a01341f23ffd3ca2ecc35e923" gracePeriod=45
Sep 10 16:04:12.516960 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.516893    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:04:12.593616 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.593575    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-model-cache\") pod \"227622f8-939d-4ff5-a22f-4662135c4ccf\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") "
Sep 10 16:04:12.593791 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.593644    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-home\") pod \"227622f8-939d-4ff5-a22f-4662135c4ccf\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") "
Sep 10 16:04:12.593791 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.593685    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-dshm\") pod \"227622f8-939d-4ff5-a22f-4662135c4ccf\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") "
Sep 10 16:04:12.593791 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.593722    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/227622f8-939d-4ff5-a22f-4662135c4ccf-tls-certs\") pod \"227622f8-939d-4ff5-a22f-4662135c4ccf\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") "
Sep 10 16:04:12.593791 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.593749    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tscnm\" (UniqueName: \"kubernetes.io/projected/227622f8-939d-4ff5-a22f-4662135c4ccf-kube-api-access-tscnm\") pod \"227622f8-939d-4ff5-a22f-4662135c4ccf\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") "
Sep 10 16:04:12.593791 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.593780    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-tmp-dir\") pod \"227622f8-939d-4ff5-a22f-4662135c4ccf\" (UID: \"227622f8-939d-4ff5-a22f-4662135c4ccf\") "
Sep 10 16:04:12.594146 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.593899    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-model-cache" (OuterVolumeSpecName: "model-cache") pod "227622f8-939d-4ff5-a22f-4662135c4ccf" (UID: "227622f8-939d-4ff5-a22f-4662135c4ccf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:12.594146 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.593961    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-home" (OuterVolumeSpecName: "home") pod "227622f8-939d-4ff5-a22f-4662135c4ccf" (UID: "227622f8-939d-4ff5-a22f-4662135c4ccf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:12.594146 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.594134    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "227622f8-939d-4ff5-a22f-4662135c4ccf" (UID: "227622f8-939d-4ff5-a22f-4662135c4ccf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:12.594340 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.594150    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-model-cache\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:12.594340 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.594167    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-home\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:12.596160 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.596134    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/227622f8-939d-4ff5-a22f-4662135c4ccf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "227622f8-939d-4ff5-a22f-4662135c4ccf" (UID: "227622f8-939d-4ff5-a22f-4662135c4ccf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:04:12.596160 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.596137    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227622f8-939d-4ff5-a22f-4662135c4ccf-kube-api-access-tscnm" (OuterVolumeSpecName: "kube-api-access-tscnm") pod "227622f8-939d-4ff5-a22f-4662135c4ccf" (UID: "227622f8-939d-4ff5-a22f-4662135c4ccf"). InnerVolumeSpecName "kube-api-access-tscnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:04:12.596318 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.596187    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-dshm" (OuterVolumeSpecName: "dshm") pod "227622f8-939d-4ff5-a22f-4662135c4ccf" (UID: "227622f8-939d-4ff5-a22f-4662135c4ccf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:12.695759 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.695713    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-dshm\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:12.695759 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.695749    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/227622f8-939d-4ff5-a22f-4662135c4ccf-tls-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:12.695759 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.695765    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tscnm\" (UniqueName: \"kubernetes.io/projected/227622f8-939d-4ff5-a22f-4662135c4ccf-kube-api-access-tscnm\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:12.696005 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:12.695777    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/227622f8-939d-4ff5-a22f-4662135c4ccf-tmp-dir\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:13.211602 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.211554    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn" podUID="57cc7f1d-14a7-4f79-81d3-bf6512daa0e7" containerName="main" containerID="cri-o://78dffccb783190380c7146d3652a68bde421f75ab0abe067e2262a51b46cd8e2" gracePeriod=45
Sep 10 16:04:13.227409 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.227372    2572 generic.go:356] "Generic (PLEG): container finished" podID="227622f8-939d-4ff5-a22f-4662135c4ccf" containerID="1ea403e3308d2130f43804e80871ba670f723e7a01341f23ffd3ca2ecc35e923" exitCode=0
Sep 10 16:04:13.227554 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.227416    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"
Sep 10 16:04:13.227554 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.227448    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x" event={"ID":"227622f8-939d-4ff5-a22f-4662135c4ccf","Type":"ContainerDied","Data":"1ea403e3308d2130f43804e80871ba670f723e7a01341f23ffd3ca2ecc35e923"}
Sep 10 16:04:13.227554 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.227496    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x" event={"ID":"227622f8-939d-4ff5-a22f-4662135c4ccf","Type":"ContainerDied","Data":"14b084b9cae173597bfbe3f8ecb24317957a450923689608cdf76ed2d1680bcf"}
Sep 10 16:04:13.227554 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.227518    2572 scope.go:117] "RemoveContainer" containerID="1ea403e3308d2130f43804e80871ba670f723e7a01341f23ffd3ca2ecc35e923"
Sep 10 16:04:13.285289 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.285257    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"]
Sep 10 16:04:13.289630 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.289596    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-50ed73/auth-test-model-50ed73-kserve-54fc8bb4c5-vpm7x"]
Sep 10 16:04:13.325216 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.325114    2572 scope.go:117] "RemoveContainer" containerID="1ea403e3308d2130f43804e80871ba670f723e7a01341f23ffd3ca2ecc35e923"
Sep 10 16:04:13.325586 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:04:13.325555    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ea403e3308d2130f43804e80871ba670f723e7a01341f23ffd3ca2ecc35e923\": container with ID starting with 1ea403e3308d2130f43804e80871ba670f723e7a01341f23ffd3ca2ecc35e923 not found: ID does not exist" containerID="1ea403e3308d2130f43804e80871ba670f723e7a01341f23ffd3ca2ecc35e923"
Sep 10 16:04:13.325670 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.325598    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ea403e3308d2130f43804e80871ba670f723e7a01341f23ffd3ca2ecc35e923"} err="failed to get container status \"1ea403e3308d2130f43804e80871ba670f723e7a01341f23ffd3ca2ecc35e923\": rpc error: code = NotFound desc = could not find container \"1ea403e3308d2130f43804e80871ba670f723e7a01341f23ffd3ca2ecc35e923\": container with ID starting with 1ea403e3308d2130f43804e80871ba670f723e7a01341f23ffd3ca2ecc35e923 not found: ID does not exist"
Sep 10 16:04:13.465318 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.465230    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:04:13.605227 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.605188    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-home\") pod \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") "
Sep 10 16:04:13.605227 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.605233    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-tls-certs\") pod \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") "
Sep 10 16:04:13.605644 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.605258    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-model-cache\") pod \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") "
Sep 10 16:04:13.605644 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.605332    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-dshm\") pod \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") "
Sep 10 16:04:13.605644 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.605398    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-tmp-dir\") pod \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") "
Sep 10 16:04:13.605644 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.605417    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9spkc\" (UniqueName: \"kubernetes.io/projected/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-kube-api-access-9spkc\") pod \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\" (UID: \"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7\") "
Sep 10 16:04:13.605644 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.605592    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "57cc7f1d-14a7-4f79-81d3-bf6512daa0e7" (UID: "57cc7f1d-14a7-4f79-81d3-bf6512daa0e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:13.605897 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.605656    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-home" (OuterVolumeSpecName: "home") pod "57cc7f1d-14a7-4f79-81d3-bf6512daa0e7" (UID: "57cc7f1d-14a7-4f79-81d3-bf6512daa0e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:13.605897 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.605719    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57cc7f1d-14a7-4f79-81d3-bf6512daa0e7" (UID: "57cc7f1d-14a7-4f79-81d3-bf6512daa0e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:13.605897 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.605823    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-tmp-dir\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:13.605897 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.605843    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-home\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:13.605897 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.605857    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-model-cache\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:13.607796 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.607773    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-kube-api-access-9spkc" (OuterVolumeSpecName: "kube-api-access-9spkc") pod "57cc7f1d-14a7-4f79-81d3-bf6512daa0e7" (UID: "57cc7f1d-14a7-4f79-81d3-bf6512daa0e7"). InnerVolumeSpecName "kube-api-access-9spkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:04:13.607974 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.607951    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-dshm" (OuterVolumeSpecName: "dshm") pod "57cc7f1d-14a7-4f79-81d3-bf6512daa0e7" (UID: "57cc7f1d-14a7-4f79-81d3-bf6512daa0e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:13.608080 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.607974    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57cc7f1d-14a7-4f79-81d3-bf6512daa0e7" (UID: "57cc7f1d-14a7-4f79-81d3-bf6512daa0e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:04:13.707202 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.707164    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-dshm\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:13.707202 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.707196    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9spkc\" (UniqueName: \"kubernetes.io/projected/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-kube-api-access-9spkc\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:13.707202 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.707208    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7-tls-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:13.996362 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:13.996324    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="227622f8-939d-4ff5-a22f-4662135c4ccf" path="/var/lib/kubelet/pods/227622f8-939d-4ff5-a22f-4662135c4ccf/volumes"
Sep 10 16:04:14.235321 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:14.235265    2572 generic.go:356] "Generic (PLEG): container finished" podID="57cc7f1d-14a7-4f79-81d3-bf6512daa0e7" containerID="78dffccb783190380c7146d3652a68bde421f75ab0abe067e2262a51b46cd8e2" exitCode=0
Sep 10 16:04:14.235494 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:14.235327    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"
Sep 10 16:04:14.235494 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:14.235351    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn" event={"ID":"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7","Type":"ContainerDied","Data":"78dffccb783190380c7146d3652a68bde421f75ab0abe067e2262a51b46cd8e2"}
Sep 10 16:04:14.235494 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:14.235391    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn" event={"ID":"57cc7f1d-14a7-4f79-81d3-bf6512daa0e7","Type":"ContainerDied","Data":"3848b1fd00ca52355fe7e1b58f3137d6e97fefa89f91f67eb71385809a191a93"}
Sep 10 16:04:14.235494 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:14.235408    2572 scope.go:117] "RemoveContainer" containerID="78dffccb783190380c7146d3652a68bde421f75ab0abe067e2262a51b46cd8e2"
Sep 10 16:04:14.245073 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:14.245051    2572 scope.go:117] "RemoveContainer" containerID="78dffccb783190380c7146d3652a68bde421f75ab0abe067e2262a51b46cd8e2"
Sep 10 16:04:14.245395 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:04:14.245373    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78dffccb783190380c7146d3652a68bde421f75ab0abe067e2262a51b46cd8e2\": container with ID starting with 78dffccb783190380c7146d3652a68bde421f75ab0abe067e2262a51b46cd8e2 not found: ID does not exist" containerID="78dffccb783190380c7146d3652a68bde421f75ab0abe067e2262a51b46cd8e2"
Sep 10 16:04:14.245460 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:14.245404    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78dffccb783190380c7146d3652a68bde421f75ab0abe067e2262a51b46cd8e2"} err="failed to get container status \"78dffccb783190380c7146d3652a68bde421f75ab0abe067e2262a51b46cd8e2\": rpc error: code = NotFound desc = could not find container \"78dffccb783190380c7146d3652a68bde421f75ab0abe067e2262a51b46cd8e2\": container with ID starting with 78dffccb783190380c7146d3652a68bde421f75ab0abe067e2262a51b46cd8e2 not found: ID does not exist"
Sep 10 16:04:14.255788 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:14.255713    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"]
Sep 10 16:04:14.258735 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:14.258703    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67df2b/auth-test-model-67df2b-kserve-d5b5c7c9f-bblwn"]
Sep 10 16:04:15.992392 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:15.992347    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57cc7f1d-14a7-4f79-81d3-bf6512daa0e7" path="/var/lib/kubelet/pods/57cc7f1d-14a7-4f79-81d3-bf6512daa0e7/volumes"
Sep 10 16:04:19.994346 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:19.994295    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:19.994916 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:19.994894    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:20.011259 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:20.011225    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:20.284353 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:20.284248    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:04:48.994770 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:48.994684    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"]
Sep 10 16:04:48.995379 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:48.995241    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57cc7f1d-14a7-4f79-81d3-bf6512daa0e7" containerName="main"
Sep 10 16:04:48.995379 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:48.995261    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="57cc7f1d-14a7-4f79-81d3-bf6512daa0e7" containerName="main"
Sep 10 16:04:48.995379 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:48.995281    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="227622f8-939d-4ff5-a22f-4662135c4ccf" containerName="main"
Sep 10 16:04:48.995379 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:48.995289    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="227622f8-939d-4ff5-a22f-4662135c4ccf" containerName="main"
Sep 10 16:04:48.995610 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:48.995474    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="57cc7f1d-14a7-4f79-81d3-bf6512daa0e7" containerName="main"
Sep 10 16:04:48.995610 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:48.995489    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="227622f8-939d-4ff5-a22f-4662135c4ccf" containerName="main"
Sep 10 16:04:48.998666 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:48.998635    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.016758 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.016734    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-67df2b\"/\"openshift-service-ca.crt\""
Sep 10 16:04:49.016903 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.016770    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-67df2b\"/\"kube-root-ca.crt\""
Sep 10 16:04:49.016903 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.016738    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-67df2b\"/\"sub-test-model-67df2b-kserve-self-signed-certs\""
Sep 10 16:04:49.017008 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.016894    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-67df2b\"/\"default-dockercfg-6bf8n\""
Sep 10 16:04:49.023321 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.023274    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"]
Sep 10 16:04:49.060250 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.060203    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-home\") pod \"sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.060250 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.060254    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-dshm\") pod \"sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.060540 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.060360    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz6xx\" (UniqueName: \"kubernetes.io/projected/1d070387-3f9b-4da3-b58b-adfb5511956a-kube-api-access-fz6xx\") pod \"sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.060540 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.060412    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-tmp-dir\") pod \"sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.060540 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.060451    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-model-cache\") pod \"sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.060540 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.060504    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d070387-3f9b-4da3-b58b-adfb5511956a-tls-certs\") pod \"sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.161920 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.161878    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-home\") pod \"sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.162153 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.161931    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-dshm\") pod \"sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.162153 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.161974    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz6xx\" (UniqueName: \"kubernetes.io/projected/1d070387-3f9b-4da3-b58b-adfb5511956a-kube-api-access-fz6xx\") pod \"sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.162153 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.162004    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-tmp-dir\") pod \"sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.162153 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.162030    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-model-cache\") pod \"sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.162153 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.162072    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d070387-3f9b-4da3-b58b-adfb5511956a-tls-certs\") pod \"sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.162494 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.162464    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-tmp-dir\") pod \"sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.162541 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.162473    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-home\") pod \"sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.162697 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.162669    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-model-cache\") pod \"sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.164445 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.164425    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-dshm\") pod \"sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.164707 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.164690    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d070387-3f9b-4da3-b58b-adfb5511956a-tls-certs\") pod \"sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.172227 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.172197    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz6xx\" (UniqueName: \"kubernetes.io/projected/1d070387-3f9b-4da3-b58b-adfb5511956a-kube-api-access-fz6xx\") pod \"sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.307988 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.307947    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:49.601409 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:04:49.601376    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d070387_3f9b_4da3_b58b_adfb5511956a.slice/crio-b14e93fef2faaf3dbae159f4535f1d661b148c85ff22028a9ddd064a66e97a7a WatchSource:0}: Error finding container b14e93fef2faaf3dbae159f4535f1d661b148c85ff22028a9ddd064a66e97a7a: Status 404 returned error can't find the container with id b14e93fef2faaf3dbae159f4535f1d661b148c85ff22028a9ddd064a66e97a7a
Sep 10 16:04:49.602829 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:49.602802    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"]
Sep 10 16:04:50.448697 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:50.448649    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8" event={"ID":"1d070387-3f9b-4da3-b58b-adfb5511956a","Type":"ContainerStarted","Data":"6ac7f853f1a85b1302e032bdaf0da2c8371e7d8284c801efaf57b9ee716aed99"}
Sep 10 16:04:50.448697 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:50.448699    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8" event={"ID":"1d070387-3f9b-4da3-b58b-adfb5511956a","Type":"ContainerStarted","Data":"b14e93fef2faaf3dbae159f4535f1d661b148c85ff22028a9ddd064a66e97a7a"}
Sep 10 16:04:50.473909 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:50.473856    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8" podStartSLOduration=2.473839728 podStartE2EDuration="2.473839728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:04:50.473795704 +0000 UTC m=+1363.146205787" watchObservedRunningTime="2026-09-10 16:04:50.473839728 +0000 UTC m=+1363.146249810"
Sep 10 16:04:51.970456 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:51.970422    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"]
Sep 10 16:04:51.970902 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:51.970643    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2" podUID="b475aee9-ec85-45a1-8167-5b83ff12bfdf" containerName="istio-proxy" containerID="cri-o://2fa50ee3673bd7329d42142802b7184cea259e991b099f296825173db1c48403" gracePeriod=30
Sep 10 16:04:54.194081 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:54.194042    2572 patch_prober.go:28] interesting pod/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 16:04:54.194485 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:54.194100    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2" podUID="b475aee9-ec85-45a1-8167-5b83ff12bfdf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 16:04:57.221784 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.221752    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:04:57.347521 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.347485    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-data\") pod \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") "
Sep 10 16:04:57.347724 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.347544    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-podinfo\") pod \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") "
Sep 10 16:04:57.347724 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.347596    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istiod-ca-cert\") pod \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") "
Sep 10 16:04:57.347724 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.347618    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-envoy\") pod \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") "
Sep 10 16:04:57.347724 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.347640    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-workload-certs\") pod \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") "
Sep 10 16:04:57.347724 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.347688    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-workload-socket\") pod \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") "
Sep 10 16:04:57.347724 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.347724    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-token\") pod \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") "
Sep 10 16:04:57.348056 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.347755    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-credential-socket\") pod \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") "
Sep 10 16:04:57.348056 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.347825    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kfpp\" (UniqueName: \"kubernetes.io/projected/b475aee9-ec85-45a1-8167-5b83ff12bfdf-kube-api-access-7kfpp\") pod \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\" (UID: \"b475aee9-ec85-45a1-8167-5b83ff12bfdf\") "
Sep 10 16:04:57.348056 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.347849    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-data" (OuterVolumeSpecName: "istio-data") pod "b475aee9-ec85-45a1-8167-5b83ff12bfdf" (UID: "b475aee9-ec85-45a1-8167-5b83ff12bfdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:57.348056 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.347968    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b475aee9-ec85-45a1-8167-5b83ff12bfdf" (UID: "b475aee9-ec85-45a1-8167-5b83ff12bfdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:57.348056 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.348010    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b475aee9-ec85-45a1-8167-5b83ff12bfdf" (UID: "b475aee9-ec85-45a1-8167-5b83ff12bfdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:57.348280 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.348066    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b475aee9-ec85-45a1-8167-5b83ff12bfdf" (UID: "b475aee9-ec85-45a1-8167-5b83ff12bfdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:04:57.348280 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.348218    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b475aee9-ec85-45a1-8167-5b83ff12bfdf" (UID: "b475aee9-ec85-45a1-8167-5b83ff12bfdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:57.348280 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.348224    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istiod-ca-cert\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:57.348280 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.348258    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-workload-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:57.348280 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.348273    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-workload-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:57.348509 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.348286    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-data\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:57.349927 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.349903    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b475aee9-ec85-45a1-8167-5b83ff12bfdf" (UID: "b475aee9-ec85-45a1-8167-5b83ff12bfdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:04:57.350038 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.349959    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b475aee9-ec85-45a1-8167-5b83ff12bfdf" (UID: "b475aee9-ec85-45a1-8167-5b83ff12bfdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:57.350236 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.350219    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b475aee9-ec85-45a1-8167-5b83ff12bfdf-kube-api-access-7kfpp" (OuterVolumeSpecName: "kube-api-access-7kfpp") pod "b475aee9-ec85-45a1-8167-5b83ff12bfdf" (UID: "b475aee9-ec85-45a1-8167-5b83ff12bfdf"). InnerVolumeSpecName "kube-api-access-7kfpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:04:57.350236 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.350229    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-token" (OuterVolumeSpecName: "istio-token") pod "b475aee9-ec85-45a1-8167-5b83ff12bfdf" (UID: "b475aee9-ec85-45a1-8167-5b83ff12bfdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:04:57.449554 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.449516    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-token\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:57.449554 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.449548    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-credential-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:57.449554 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.449558    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kfpp\" (UniqueName: \"kubernetes.io/projected/b475aee9-ec85-45a1-8167-5b83ff12bfdf-kube-api-access-7kfpp\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:57.449554 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.449567    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-podinfo\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:57.449822 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.449575    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b475aee9-ec85-45a1-8167-5b83ff12bfdf-istio-envoy\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:04:57.493460 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.493424    2572 generic.go:356] "Generic (PLEG): container finished" podID="b475aee9-ec85-45a1-8167-5b83ff12bfdf" containerID="2fa50ee3673bd7329d42142802b7184cea259e991b099f296825173db1c48403" exitCode=0
Sep 10 16:04:57.493631 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.493481    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"
Sep 10 16:04:57.493631 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.493512    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2" event={"ID":"b475aee9-ec85-45a1-8167-5b83ff12bfdf","Type":"ContainerDied","Data":"2fa50ee3673bd7329d42142802b7184cea259e991b099f296825173db1c48403"}
Sep 10 16:04:57.493631 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.493569    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2" event={"ID":"b475aee9-ec85-45a1-8167-5b83ff12bfdf","Type":"ContainerDied","Data":"f1540c0969d04f8a39122054cdcb733e52205d9cdada58d7c0dba00cd1d08bd4"}
Sep 10 16:04:57.493631 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.493593    2572 scope.go:117] "RemoveContainer" containerID="2fa50ee3673bd7329d42142802b7184cea259e991b099f296825173db1c48403"
Sep 10 16:04:57.503729 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.503709    2572 scope.go:117] "RemoveContainer" containerID="c46d8a3081246e383c9196fd54afb06909956f78e1db0c5dd2be88a0dc2820dc"
Sep 10 16:04:57.513199 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.513173    2572 scope.go:117] "RemoveContainer" containerID="2fa50ee3673bd7329d42142802b7184cea259e991b099f296825173db1c48403"
Sep 10 16:04:57.513534 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:04:57.513514    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fa50ee3673bd7329d42142802b7184cea259e991b099f296825173db1c48403\": container with ID starting with 2fa50ee3673bd7329d42142802b7184cea259e991b099f296825173db1c48403 not found: ID does not exist" containerID="2fa50ee3673bd7329d42142802b7184cea259e991b099f296825173db1c48403"
Sep 10 16:04:57.513600 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.513543    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fa50ee3673bd7329d42142802b7184cea259e991b099f296825173db1c48403"} err="failed to get container status \"2fa50ee3673bd7329d42142802b7184cea259e991b099f296825173db1c48403\": rpc error: code = NotFound desc = could not find container \"2fa50ee3673bd7329d42142802b7184cea259e991b099f296825173db1c48403\": container with ID starting with 2fa50ee3673bd7329d42142802b7184cea259e991b099f296825173db1c48403 not found: ID does not exist"
Sep 10 16:04:57.513600 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.513561    2572 scope.go:117] "RemoveContainer" containerID="c46d8a3081246e383c9196fd54afb06909956f78e1db0c5dd2be88a0dc2820dc"
Sep 10 16:04:57.513837 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:04:57.513815    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c46d8a3081246e383c9196fd54afb06909956f78e1db0c5dd2be88a0dc2820dc\": container with ID starting with c46d8a3081246e383c9196fd54afb06909956f78e1db0c5dd2be88a0dc2820dc not found: ID does not exist" containerID="c46d8a3081246e383c9196fd54afb06909956f78e1db0c5dd2be88a0dc2820dc"
Sep 10 16:04:57.513896 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.513847    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c46d8a3081246e383c9196fd54afb06909956f78e1db0c5dd2be88a0dc2820dc"} err="failed to get container status \"c46d8a3081246e383c9196fd54afb06909956f78e1db0c5dd2be88a0dc2820dc\": rpc error: code = NotFound desc = could not find container \"c46d8a3081246e383c9196fd54afb06909956f78e1db0c5dd2be88a0dc2820dc\": container with ID starting with c46d8a3081246e383c9196fd54afb06909956f78e1db0c5dd2be88a0dc2820dc not found: ID does not exist"
Sep 10 16:04:57.522524 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.522495    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"]
Sep 10 16:04:57.530758 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.530734    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0e6e0dcc-openshift-default-5bbc98b9d8-2r5w2"]
Sep 10 16:04:57.992967 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:57.992928    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b475aee9-ec85-45a1-8167-5b83ff12bfdf" path="/var/lib/kubelet/pods/b475aee9-ec85-45a1-8167-5b83ff12bfdf/volumes"
Sep 10 16:04:59.308935 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:59.308886    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:59.308935 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:59.308945    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:59.321575 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:59.321540    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:04:59.517701 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:04:59.517669    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:05:39.699802 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:39.699764    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7595-5fc359"]
Sep 10 16:05:39.700321 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:39.700137    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b475aee9-ec85-45a1-8167-5b83ff12bfdf" containerName="istio-proxy"
Sep 10 16:05:39.700321 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:39.700150    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b475aee9-ec85-45a1-8167-5b83ff12bfdf" containerName="istio-proxy"
Sep 10 16:05:39.700321 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:39.700163    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b475aee9-ec85-45a1-8167-5b83ff12bfdf" containerName="istio-proxy"
Sep 10 16:05:39.700321 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:39.700169    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b475aee9-ec85-45a1-8167-5b83ff12bfdf" containerName="istio-proxy"
Sep 10 16:05:39.700321 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:39.700232    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b475aee9-ec85-45a1-8167-5b83ff12bfdf" containerName="istio-proxy"
Sep 10 16:05:39.700321 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:39.700240    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b475aee9-ec85-45a1-8167-5b83ff12bfdf" containerName="istio-proxy"
Sep 10 16:05:39.703666 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:39.703647    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7595-5fc359"
Sep 10 16:05:39.706504 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:39.706484    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-j2pv4\""
Sep 10 16:05:39.710553 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:39.710523    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7595-5fc359"]
Sep 10 16:05:39.868141 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:39.868094    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8md57\" (UniqueName: \"kubernetes.io/projected/8f41b288-af56-478f-b270-303f3fdaf810-kube-api-access-8md57\") pod \"test-curl-7595-5fc359\" (UID: \"8f41b288-af56-478f-b270-303f3fdaf810\") " pod="opendatahub/test-curl-7595-5fc359"
Sep 10 16:05:39.969213 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:39.969095    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8md57\" (UniqueName: \"kubernetes.io/projected/8f41b288-af56-478f-b270-303f3fdaf810-kube-api-access-8md57\") pod \"test-curl-7595-5fc359\" (UID: \"8f41b288-af56-478f-b270-303f3fdaf810\") " pod="opendatahub/test-curl-7595-5fc359"
Sep 10 16:05:39.978131 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:39.978097    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8md57\" (UniqueName: \"kubernetes.io/projected/8f41b288-af56-478f-b270-303f3fdaf810-kube-api-access-8md57\") pod \"test-curl-7595-5fc359\" (UID: \"8f41b288-af56-478f-b270-303f3fdaf810\") " pod="opendatahub/test-curl-7595-5fc359"
Sep 10 16:05:40.014147 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:40.014114    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7595-5fc359"
Sep 10 16:05:40.293224 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:40.293152    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7595-5fc359"]
Sep 10 16:05:40.296202 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:05:40.296170    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f41b288_af56_478f_b270_303f3fdaf810.slice/crio-56b0059ac784eaa3429294f406862df2a89dd2e0a365bae7aea2489d388e1abe WatchSource:0}: Error finding container 56b0059ac784eaa3429294f406862df2a89dd2e0a365bae7aea2489d388e1abe: Status 404 returned error can't find the container with id 56b0059ac784eaa3429294f406862df2a89dd2e0a365bae7aea2489d388e1abe
Sep 10 16:05:40.756795 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:40.756752    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7595-5fc359" event={"ID":"8f41b288-af56-478f-b270-303f3fdaf810","Type":"ContainerStarted","Data":"0b25db28aebd74645bfdb8a9e1040adb61173df66a6421e2a0db0535c3046ac0"}
Sep 10 16:05:40.756795 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:40.756795    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7595-5fc359" event={"ID":"8f41b288-af56-478f-b270-303f3fdaf810","Type":"ContainerStarted","Data":"56b0059ac784eaa3429294f406862df2a89dd2e0a365bae7aea2489d388e1abe"}
Sep 10 16:05:40.777889 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:40.777834    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7595-5fc359" podStartSLOduration=1.777817006 podStartE2EDuration="1.777817006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:05:40.775949409 +0000 UTC m=+1413.448359493" watchObservedRunningTime="2026-09-10 16:05:40.777817006 +0000 UTC m=+1413.450227089"
Sep 10 16:05:41.401910 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:41.401871    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7595-5fc359"]
Sep 10 16:05:41.409492 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:41.409462    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7595-5fc359"]
Sep 10 16:05:41.531163 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:41.531074    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7595-1bdba8"]
Sep 10 16:05:41.531510 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:41.531495    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f41b288-af56-478f-b270-303f3fdaf810" containerName="test-curl-7595-5fc359"
Sep 10 16:05:41.531566 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:41.531512    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f41b288-af56-478f-b270-303f3fdaf810" containerName="test-curl-7595-5fc359"
Sep 10 16:05:41.531608 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:41.531568    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f41b288-af56-478f-b270-303f3fdaf810" containerName="test-curl-7595-5fc359"
Sep 10 16:05:41.536076 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:41.536043    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7595-1bdba8"
Sep 10 16:05:41.544391 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:41.544356    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7595-1bdba8"]
Sep 10 16:05:41.584641 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:41.584604    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh2xh\" (UniqueName: \"kubernetes.io/projected/a24c2924-fae3-4ec5-b8ec-2a98de7b997d-kube-api-access-bh2xh\") pod \"test-curl-7595-1bdba8\" (UID: \"a24c2924-fae3-4ec5-b8ec-2a98de7b997d\") " pod="opendatahub/test-curl-7595-1bdba8"
Sep 10 16:05:41.685159 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:41.685116    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh2xh\" (UniqueName: \"kubernetes.io/projected/a24c2924-fae3-4ec5-b8ec-2a98de7b997d-kube-api-access-bh2xh\") pod \"test-curl-7595-1bdba8\" (UID: \"a24c2924-fae3-4ec5-b8ec-2a98de7b997d\") " pod="opendatahub/test-curl-7595-1bdba8"
Sep 10 16:05:41.693992 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:41.693953    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh2xh\" (UniqueName: \"kubernetes.io/projected/a24c2924-fae3-4ec5-b8ec-2a98de7b997d-kube-api-access-bh2xh\") pod \"test-curl-7595-1bdba8\" (UID: \"a24c2924-fae3-4ec5-b8ec-2a98de7b997d\") " pod="opendatahub/test-curl-7595-1bdba8"
Sep 10 16:05:41.762960 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:05:41.762928    2572 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7595-5fc359" container="test-curl-7595-5fc359"
Sep 10 16:05:41.849954 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:41.849913    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7595-1bdba8"
Sep 10 16:05:42.124017 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:42.123981    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7595-1bdba8"]
Sep 10 16:05:42.124729 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:05:42.124704    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda24c2924_fae3_4ec5_b8ec_2a98de7b997d.slice/crio-a1be7edd98ac558cfa27583736e865200ad8c00ba3121781619974f051dde946 WatchSource:0}: Error finding container a1be7edd98ac558cfa27583736e865200ad8c00ba3121781619974f051dde946: Status 404 returned error can't find the container with id a1be7edd98ac558cfa27583736e865200ad8c00ba3121781619974f051dde946
Sep 10 16:05:42.770418 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:42.770373    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7595-1bdba8" event={"ID":"a24c2924-fae3-4ec5-b8ec-2a98de7b997d","Type":"ContainerStarted","Data":"bf747cb093b1d4a12b49183f77003f93e112533f1b37eb07c3e37eb9d3e17073"}
Sep 10 16:05:42.770945 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:42.770428    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7595-1bdba8" event={"ID":"a24c2924-fae3-4ec5-b8ec-2a98de7b997d","Type":"ContainerStarted","Data":"a1be7edd98ac558cfa27583736e865200ad8c00ba3121781619974f051dde946"}
Sep 10 16:05:42.770945 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:42.770537    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7595-5fc359" podUID="8f41b288-af56-478f-b270-303f3fdaf810" containerName="test-curl-7595-5fc359" containerID="cri-o://0b25db28aebd74645bfdb8a9e1040adb61173df66a6421e2a0db0535c3046ac0" gracePeriod=2
Sep 10 16:05:42.785580 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:42.785530    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7595-1bdba8" podStartSLOduration=1.785509708 podStartE2EDuration="1.785509708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:05:42.784975387 +0000 UTC m=+1415.457385470" watchObservedRunningTime="2026-09-10 16:05:42.785509708 +0000 UTC m=+1415.457919769"
Sep 10 16:05:43.421898 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:43.421857    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7595-1bdba8"]
Sep 10 16:05:43.427472 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:43.427430    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7595-1bdba8"]
Sep 10 16:05:43.776173 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:05:43.776140    2572 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7595-1bdba8" container="test-curl-7595-1bdba8"
Sep 10 16:05:44.782419 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:44.782381    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7595-1bdba8" podUID="a24c2924-fae3-4ec5-b8ec-2a98de7b997d" containerName="test-curl-7595-1bdba8" containerID="cri-o://bf747cb093b1d4a12b49183f77003f93e112533f1b37eb07c3e37eb9d3e17073" gracePeriod=2
Sep 10 16:05:44.925969 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:44.925940    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7595-5fc359"
Sep 10 16:05:45.015863 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:45.015823    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8md57\" (UniqueName: \"kubernetes.io/projected/8f41b288-af56-478f-b270-303f3fdaf810-kube-api-access-8md57\") pod \"8f41b288-af56-478f-b270-303f3fdaf810\" (UID: \"8f41b288-af56-478f-b270-303f3fdaf810\") "
Sep 10 16:05:45.017982 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:45.017944    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f41b288-af56-478f-b270-303f3fdaf810-kube-api-access-8md57" (OuterVolumeSpecName: "kube-api-access-8md57") pod "8f41b288-af56-478f-b270-303f3fdaf810" (UID: "8f41b288-af56-478f-b270-303f3fdaf810"). InnerVolumeSpecName "kube-api-access-8md57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:05:45.117321 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:45.117256    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8md57\" (UniqueName: \"kubernetes.io/projected/8f41b288-af56-478f-b270-303f3fdaf810-kube-api-access-8md57\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:05:45.463560 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:45.463526    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"]
Sep 10 16:05:45.789715 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:45.789626    2572 generic.go:356] "Generic (PLEG): container finished" podID="8f41b288-af56-478f-b270-303f3fdaf810" containerID="0b25db28aebd74645bfdb8a9e1040adb61173df66a6421e2a0db0535c3046ac0" exitCode=137
Sep 10 16:05:45.789715 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:45.789668    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7595-5fc359"
Sep 10 16:05:45.789715 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:45.789715    2572 scope.go:117] "RemoveContainer" containerID="0b25db28aebd74645bfdb8a9e1040adb61173df66a6421e2a0db0535c3046ac0"
Sep 10 16:05:45.799008 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:45.798984    2572 scope.go:117] "RemoveContainer" containerID="0b25db28aebd74645bfdb8a9e1040adb61173df66a6421e2a0db0535c3046ac0"
Sep 10 16:05:45.799330 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:05:45.799286    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b25db28aebd74645bfdb8a9e1040adb61173df66a6421e2a0db0535c3046ac0\": container with ID starting with 0b25db28aebd74645bfdb8a9e1040adb61173df66a6421e2a0db0535c3046ac0 not found: ID does not exist" containerID="0b25db28aebd74645bfdb8a9e1040adb61173df66a6421e2a0db0535c3046ac0"
Sep 10 16:05:45.799379 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:45.799334    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b25db28aebd74645bfdb8a9e1040adb61173df66a6421e2a0db0535c3046ac0"} err="failed to get container status \"0b25db28aebd74645bfdb8a9e1040adb61173df66a6421e2a0db0535c3046ac0\": rpc error: code = NotFound desc = could not find container \"0b25db28aebd74645bfdb8a9e1040adb61173df66a6421e2a0db0535c3046ac0\": container with ID starting with 0b25db28aebd74645bfdb8a9e1040adb61173df66a6421e2a0db0535c3046ac0 not found: ID does not exist"
Sep 10 16:05:45.800043 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:05:45.800021    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7595-5fc359\" is forbidden: User \"system:node:ip-10-0-134-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-36.ec2.internal' and this object" podUID="8f41b288-af56-478f-b270-303f3fdaf810" pod="opendatahub/test-curl-7595-5fc359"
Sep 10 16:05:45.993043 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:45.992999    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f41b288-af56-478f-b270-303f3fdaf810" path="/var/lib/kubelet/pods/8f41b288-af56-478f-b270-303f3fdaf810/volumes"
Sep 10 16:05:46.350565 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:46.350526    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"]
Sep 10 16:05:46.952602 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:46.952570    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7595-1bdba8"
Sep 10 16:05:47.037164 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:47.037065    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh2xh\" (UniqueName: \"kubernetes.io/projected/a24c2924-fae3-4ec5-b8ec-2a98de7b997d-kube-api-access-bh2xh\") pod \"a24c2924-fae3-4ec5-b8ec-2a98de7b997d\" (UID: \"a24c2924-fae3-4ec5-b8ec-2a98de7b997d\") "
Sep 10 16:05:47.039227 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:47.039196    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24c2924-fae3-4ec5-b8ec-2a98de7b997d-kube-api-access-bh2xh" (OuterVolumeSpecName: "kube-api-access-bh2xh") pod "a24c2924-fae3-4ec5-b8ec-2a98de7b997d" (UID: "a24c2924-fae3-4ec5-b8ec-2a98de7b997d"). InnerVolumeSpecName "kube-api-access-bh2xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:05:47.138084 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:47.138042    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh2xh\" (UniqueName: \"kubernetes.io/projected/a24c2924-fae3-4ec5-b8ec-2a98de7b997d-kube-api-access-bh2xh\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:05:47.805377 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:47.805323    2572 generic.go:356] "Generic (PLEG): container finished" podID="a24c2924-fae3-4ec5-b8ec-2a98de7b997d" containerID="bf747cb093b1d4a12b49183f77003f93e112533f1b37eb07c3e37eb9d3e17073" exitCode=137
Sep 10 16:05:47.805576 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:47.805392    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7595-1bdba8"
Sep 10 16:05:47.805576 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:47.805411    2572 scope.go:117] "RemoveContainer" containerID="bf747cb093b1d4a12b49183f77003f93e112533f1b37eb07c3e37eb9d3e17073"
Sep 10 16:05:47.815109 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:47.814592    2572 scope.go:117] "RemoveContainer" containerID="bf747cb093b1d4a12b49183f77003f93e112533f1b37eb07c3e37eb9d3e17073"
Sep 10 16:05:47.815109 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:05:47.814914    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf747cb093b1d4a12b49183f77003f93e112533f1b37eb07c3e37eb9d3e17073\": container with ID starting with bf747cb093b1d4a12b49183f77003f93e112533f1b37eb07c3e37eb9d3e17073 not found: ID does not exist" containerID="bf747cb093b1d4a12b49183f77003f93e112533f1b37eb07c3e37eb9d3e17073"
Sep 10 16:05:47.815109 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:47.814951    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf747cb093b1d4a12b49183f77003f93e112533f1b37eb07c3e37eb9d3e17073"} err="failed to get container status \"bf747cb093b1d4a12b49183f77003f93e112533f1b37eb07c3e37eb9d3e17073\": rpc error: code = NotFound desc = could not find container \"bf747cb093b1d4a12b49183f77003f93e112533f1b37eb07c3e37eb9d3e17073\": container with ID starting with bf747cb093b1d4a12b49183f77003f93e112533f1b37eb07c3e37eb9d3e17073 not found: ID does not exist"
Sep 10 16:05:47.817194 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:05:47.817167    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7595-1bdba8\" is forbidden: User \"system:node:ip-10-0-134-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-36.ec2.internal' and this object" podUID="a24c2924-fae3-4ec5-b8ec-2a98de7b997d" pod="opendatahub/test-curl-7595-1bdba8"
Sep 10 16:05:47.991033 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:05:47.990990    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7595-1bdba8\" is forbidden: User \"system:node:ip-10-0-134-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-36.ec2.internal' and this object" podUID="a24c2924-fae3-4ec5-b8ec-2a98de7b997d" pod="opendatahub/test-curl-7595-1bdba8"
Sep 10 16:05:47.992777 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:47.992755    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a24c2924-fae3-4ec5-b8ec-2a98de7b997d" path="/var/lib/kubelet/pods/a24c2924-fae3-4ec5-b8ec-2a98de7b997d/volumes"
Sep 10 16:05:48.983094 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:48.983056    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"]
Sep 10 16:05:48.983529 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:48.983514    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a24c2924-fae3-4ec5-b8ec-2a98de7b997d" containerName="test-curl-7595-1bdba8"
Sep 10 16:05:48.983598 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:48.983531    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a24c2924-fae3-4ec5-b8ec-2a98de7b997d" containerName="test-curl-7595-1bdba8"
Sep 10 16:05:48.983659 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:48.983608    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a24c2924-fae3-4ec5-b8ec-2a98de7b997d" containerName="test-curl-7595-1bdba8"
Sep 10 16:05:48.988381 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:48.988353    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:48.991494 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:48.991469    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-45e544-openshift-default-dockercfg-v4v8v\""
Sep 10 16:05:49.009257 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.009223    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"]
Sep 10 16:05:49.057094 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.057043    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5810430-84a6-4e80-a3c0-43767806dbf0-istiod-ca-cert\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.057464 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.057100    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-data\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.057464 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.057211    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-credential-socket\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.057464 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.057277    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-envoy\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.057464 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.057338    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnt4w\" (UniqueName: \"kubernetes.io/projected/a5810430-84a6-4e80-a3c0-43767806dbf0-kube-api-access-rnt4w\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.057464 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.057421    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-workload-socket\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.057784 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.057497    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-token\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.057784 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.057523    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-workload-certs\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.057784 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.057560    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-podinfo\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.158515 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.158467    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnt4w\" (UniqueName: \"kubernetes.io/projected/a5810430-84a6-4e80-a3c0-43767806dbf0-kube-api-access-rnt4w\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.158515 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.158525    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-workload-socket\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.158821 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.158699    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-token\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.158821 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.158744    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-workload-certs\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.158821 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.158778    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-podinfo\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.158981 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.158836    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5810430-84a6-4e80-a3c0-43767806dbf0-istiod-ca-cert\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.158981 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.158880    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-data\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.158981 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.158942    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-credential-socket\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.158981 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.158879    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-workload-socket\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.159195 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.159023    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-envoy\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.159195 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.159100    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-data\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.159340 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.159214    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-workload-certs\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.159340 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.159288    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-credential-socket\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.159535 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.159508    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5810430-84a6-4e80-a3c0-43767806dbf0-istiod-ca-cert\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.161331 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.161292    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-envoy\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.161458 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.161433    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-podinfo\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.171877 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.171847    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-token\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.171989 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.171920    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnt4w\" (UniqueName: \"kubernetes.io/projected/a5810430-84a6-4e80-a3c0-43767806dbf0-kube-api-access-rnt4w\") pod \"e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") " pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.299151 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.299110    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:49.594097 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.594067    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"]
Sep 10 16:05:49.596903 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:05:49.596875    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5810430_84a6_4e80_a3c0_43767806dbf0.slice/crio-0bd580a815f9ca924566baab3c4bb64cf611723714c621bf1ad857be441a56c0 WatchSource:0}: Error finding container 0bd580a815f9ca924566baab3c4bb64cf611723714c621bf1ad857be441a56c0: Status 404 returned error can't find the container with id 0bd580a815f9ca924566baab3c4bb64cf611723714c621bf1ad857be441a56c0
Sep 10 16:05:49.599409 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.599376    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:05:49.599488 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.599459    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:05:49.599531 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.599490    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:05:49.820112 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.820075    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd" event={"ID":"a5810430-84a6-4e80-a3c0-43767806dbf0","Type":"ContainerStarted","Data":"956c0e1eda1211d3fd03016ea27a12237d99b805d1a1272b715f582e73534d0d"}
Sep 10 16:05:49.820357 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.820133    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd" event={"ID":"a5810430-84a6-4e80-a3c0-43767806dbf0","Type":"ContainerStarted","Data":"0bd580a815f9ca924566baab3c4bb64cf611723714c621bf1ad857be441a56c0"}
Sep 10 16:05:49.846463 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:49.846335    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd" podStartSLOduration=1.846311748 podStartE2EDuration="1.846311748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:05:49.842590083 +0000 UTC m=+1422.515000207" watchObservedRunningTime="2026-09-10 16:05:49.846311748 +0000 UTC m=+1422.518721826"
Sep 10 16:05:50.299782 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:50.299741    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:50.301445 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:50.301409    2572 patch_prober.go:28] interesting pod/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 10 16:05:50.301578 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:50.301469    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd" podUID="a5810430-84a6-4e80-a3c0-43767806dbf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 10 16:05:51.303903 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:51.303862    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:51.833810 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:51.833765    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:05:51.834712 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:05:51.834684    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:06:00.483286 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.483176    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm" podUID="36689670-76ce-4696-96d8-2e238c562ab3" containerName="main" containerID="cri-o://737fd6a77a8939299c171962121429320c4d56e6218acae5dbc4d9b1e0551f95" gracePeriod=45
Sep 10 16:06:00.774596 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.774558    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:06:00.880987 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.880946    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-model-cache\") pod \"36689670-76ce-4696-96d8-2e238c562ab3\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") "
Sep 10 16:06:00.880987 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.880995    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-dshm\") pod \"36689670-76ce-4696-96d8-2e238c562ab3\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") "
Sep 10 16:06:00.881264 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.881041    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36689670-76ce-4696-96d8-2e238c562ab3-tls-certs\") pod \"36689670-76ce-4696-96d8-2e238c562ab3\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") "
Sep 10 16:06:00.881264 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.881064    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kz4mh\" (UniqueName: \"kubernetes.io/projected/36689670-76ce-4696-96d8-2e238c562ab3-kube-api-access-kz4mh\") pod \"36689670-76ce-4696-96d8-2e238c562ab3\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") "
Sep 10 16:06:00.881264 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.881143    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-tmp-dir\") pod \"36689670-76ce-4696-96d8-2e238c562ab3\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") "
Sep 10 16:06:00.881264 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.881187    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-home\") pod \"36689670-76ce-4696-96d8-2e238c562ab3\" (UID: \"36689670-76ce-4696-96d8-2e238c562ab3\") "
Sep 10 16:06:00.881492 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.881283    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-model-cache" (OuterVolumeSpecName: "model-cache") pod "36689670-76ce-4696-96d8-2e238c562ab3" (UID: "36689670-76ce-4696-96d8-2e238c562ab3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:00.881492 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.881429    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "36689670-76ce-4696-96d8-2e238c562ab3" (UID: "36689670-76ce-4696-96d8-2e238c562ab3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:00.881492 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.881458    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-home" (OuterVolumeSpecName: "home") pod "36689670-76ce-4696-96d8-2e238c562ab3" (UID: "36689670-76ce-4696-96d8-2e238c562ab3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:00.881657 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.881614    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-model-cache\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:06:00.881657 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.881632    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-tmp-dir\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:06:00.881657 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.881644    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-home\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:06:00.883375 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.883350    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36689670-76ce-4696-96d8-2e238c562ab3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "36689670-76ce-4696-96d8-2e238c562ab3" (UID: "36689670-76ce-4696-96d8-2e238c562ab3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:06:00.883497 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.883393    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-dshm" (OuterVolumeSpecName: "dshm") pod "36689670-76ce-4696-96d8-2e238c562ab3" (UID: "36689670-76ce-4696-96d8-2e238c562ab3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:00.883497 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.883424    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36689670-76ce-4696-96d8-2e238c562ab3-kube-api-access-kz4mh" (OuterVolumeSpecName: "kube-api-access-kz4mh") pod "36689670-76ce-4696-96d8-2e238c562ab3" (UID: "36689670-76ce-4696-96d8-2e238c562ab3"). InnerVolumeSpecName "kube-api-access-kz4mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:06:00.891149 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.891117    2572 generic.go:356] "Generic (PLEG): container finished" podID="36689670-76ce-4696-96d8-2e238c562ab3" containerID="737fd6a77a8939299c171962121429320c4d56e6218acae5dbc4d9b1e0551f95" exitCode=0
Sep 10 16:06:00.891320 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.891171    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"
Sep 10 16:06:00.891320 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.891211    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm" event={"ID":"36689670-76ce-4696-96d8-2e238c562ab3","Type":"ContainerDied","Data":"737fd6a77a8939299c171962121429320c4d56e6218acae5dbc4d9b1e0551f95"}
Sep 10 16:06:00.891320 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.891268    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm" event={"ID":"36689670-76ce-4696-96d8-2e238c562ab3","Type":"ContainerDied","Data":"89810ebed634192561b943a81c6a1fcb763b253429871b9104f0649dcda78a0a"}
Sep 10 16:06:00.891320 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.891291    2572 scope.go:117] "RemoveContainer" containerID="737fd6a77a8939299c171962121429320c4d56e6218acae5dbc4d9b1e0551f95"
Sep 10 16:06:00.903752 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.903730    2572 scope.go:117] "RemoveContainer" containerID="737fd6a77a8939299c171962121429320c4d56e6218acae5dbc4d9b1e0551f95"
Sep 10 16:06:00.904109 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:06:00.904087    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"737fd6a77a8939299c171962121429320c4d56e6218acae5dbc4d9b1e0551f95\": container with ID starting with 737fd6a77a8939299c171962121429320c4d56e6218acae5dbc4d9b1e0551f95 not found: ID does not exist" containerID="737fd6a77a8939299c171962121429320c4d56e6218acae5dbc4d9b1e0551f95"
Sep 10 16:06:00.904161 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.904120    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"737fd6a77a8939299c171962121429320c4d56e6218acae5dbc4d9b1e0551f95"} err="failed to get container status \"737fd6a77a8939299c171962121429320c4d56e6218acae5dbc4d9b1e0551f95\": rpc error: code = NotFound desc = could not find container \"737fd6a77a8939299c171962121429320c4d56e6218acae5dbc4d9b1e0551f95\": container with ID starting with 737fd6a77a8939299c171962121429320c4d56e6218acae5dbc4d9b1e0551f95 not found: ID does not exist"
Sep 10 16:06:00.915255 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.915218    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"]
Sep 10 16:06:00.936329 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.935611    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-50ed73/sub-test-model-50ed73-kserve-f4fd64795-bl5bm"]
Sep 10 16:06:00.982206 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.982171    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36689670-76ce-4696-96d8-2e238c562ab3-dshm\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:06:00.982206 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.982202    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36689670-76ce-4696-96d8-2e238c562ab3-tls-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:06:00.982206 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:00.982213    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz4mh\" (UniqueName: \"kubernetes.io/projected/36689670-76ce-4696-96d8-2e238c562ab3-kube-api-access-kz4mh\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:06:01.371074 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.371025    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8" podUID="1d070387-3f9b-4da3-b58b-adfb5511956a" containerName="main" containerID="cri-o://6ac7f853f1a85b1302e032bdaf0da2c8371e7d8284c801efaf57b9ee716aed99" gracePeriod=45
Sep 10 16:06:01.669148 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.669116    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:06:01.789463 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.789417    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-tmp-dir\") pod \"1d070387-3f9b-4da3-b58b-adfb5511956a\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") "
Sep 10 16:06:01.789642 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.789517    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz6xx\" (UniqueName: \"kubernetes.io/projected/1d070387-3f9b-4da3-b58b-adfb5511956a-kube-api-access-fz6xx\") pod \"1d070387-3f9b-4da3-b58b-adfb5511956a\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") "
Sep 10 16:06:01.789642 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.789558    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d070387-3f9b-4da3-b58b-adfb5511956a-tls-certs\") pod \"1d070387-3f9b-4da3-b58b-adfb5511956a\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") "
Sep 10 16:06:01.789642 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.789614    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-home\") pod \"1d070387-3f9b-4da3-b58b-adfb5511956a\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") "
Sep 10 16:06:01.789780 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.789648    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-dshm\") pod \"1d070387-3f9b-4da3-b58b-adfb5511956a\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") "
Sep 10 16:06:01.789780 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.789705    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d070387-3f9b-4da3-b58b-adfb5511956a" (UID: "1d070387-3f9b-4da3-b58b-adfb5511956a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:01.789780 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.789720    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-model-cache\") pod \"1d070387-3f9b-4da3-b58b-adfb5511956a\" (UID: \"1d070387-3f9b-4da3-b58b-adfb5511956a\") "
Sep 10 16:06:01.789921 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.789825    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-home" (OuterVolumeSpecName: "home") pod "1d070387-3f9b-4da3-b58b-adfb5511956a" (UID: "1d070387-3f9b-4da3-b58b-adfb5511956a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:01.790026 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.790002    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d070387-3f9b-4da3-b58b-adfb5511956a" (UID: "1d070387-3f9b-4da3-b58b-adfb5511956a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:01.790130 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.790114    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-model-cache\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:06:01.790191 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.790137    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-tmp-dir\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:06:01.790191 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.790150    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-home\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:06:01.791804 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.791784    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d070387-3f9b-4da3-b58b-adfb5511956a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d070387-3f9b-4da3-b58b-adfb5511956a" (UID: "1d070387-3f9b-4da3-b58b-adfb5511956a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:06:01.792188 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.792167    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-dshm" (OuterVolumeSpecName: "dshm") pod "1d070387-3f9b-4da3-b58b-adfb5511956a" (UID: "1d070387-3f9b-4da3-b58b-adfb5511956a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:01.792230 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.792186    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d070387-3f9b-4da3-b58b-adfb5511956a-kube-api-access-fz6xx" (OuterVolumeSpecName: "kube-api-access-fz6xx") pod "1d070387-3f9b-4da3-b58b-adfb5511956a" (UID: "1d070387-3f9b-4da3-b58b-adfb5511956a"). InnerVolumeSpecName "kube-api-access-fz6xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:06:01.891265 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.891217    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d070387-3f9b-4da3-b58b-adfb5511956a-dshm\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:06:01.891265 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.891254    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz6xx\" (UniqueName: \"kubernetes.io/projected/1d070387-3f9b-4da3-b58b-adfb5511956a-kube-api-access-fz6xx\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:06:01.891265 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.891266    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d070387-3f9b-4da3-b58b-adfb5511956a-tls-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:06:01.898923 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.898828    2572 generic.go:356] "Generic (PLEG): container finished" podID="1d070387-3f9b-4da3-b58b-adfb5511956a" containerID="6ac7f853f1a85b1302e032bdaf0da2c8371e7d8284c801efaf57b9ee716aed99" exitCode=0
Sep 10 16:06:01.898923 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.898868    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"
Sep 10 16:06:01.899154 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.898870    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8" event={"ID":"1d070387-3f9b-4da3-b58b-adfb5511956a","Type":"ContainerDied","Data":"6ac7f853f1a85b1302e032bdaf0da2c8371e7d8284c801efaf57b9ee716aed99"}
Sep 10 16:06:01.899154 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.898998    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8" event={"ID":"1d070387-3f9b-4da3-b58b-adfb5511956a","Type":"ContainerDied","Data":"b14e93fef2faaf3dbae159f4535f1d661b148c85ff22028a9ddd064a66e97a7a"}
Sep 10 16:06:01.899154 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.899021    2572 scope.go:117] "RemoveContainer" containerID="6ac7f853f1a85b1302e032bdaf0da2c8371e7d8284c801efaf57b9ee716aed99"
Sep 10 16:06:01.908854 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.908830    2572 scope.go:117] "RemoveContainer" containerID="6ac7f853f1a85b1302e032bdaf0da2c8371e7d8284c801efaf57b9ee716aed99"
Sep 10 16:06:01.909194 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:06:01.909169    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ac7f853f1a85b1302e032bdaf0da2c8371e7d8284c801efaf57b9ee716aed99\": container with ID starting with 6ac7f853f1a85b1302e032bdaf0da2c8371e7d8284c801efaf57b9ee716aed99 not found: ID does not exist" containerID="6ac7f853f1a85b1302e032bdaf0da2c8371e7d8284c801efaf57b9ee716aed99"
Sep 10 16:06:01.909263 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.909209    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac7f853f1a85b1302e032bdaf0da2c8371e7d8284c801efaf57b9ee716aed99"} err="failed to get container status \"6ac7f853f1a85b1302e032bdaf0da2c8371e7d8284c801efaf57b9ee716aed99\": rpc error: code = NotFound desc = could not find container \"6ac7f853f1a85b1302e032bdaf0da2c8371e7d8284c801efaf57b9ee716aed99\": container with ID starting with 6ac7f853f1a85b1302e032bdaf0da2c8371e7d8284c801efaf57b9ee716aed99 not found: ID does not exist"
Sep 10 16:06:01.924425 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.924356    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"]
Sep 10 16:06:01.932601 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.932557    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67df2b/sub-test-model-67df2b-kserve-7fbfb6f757-6z7g8"]
Sep 10 16:06:01.992833 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.992798    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d070387-3f9b-4da3-b58b-adfb5511956a" path="/var/lib/kubelet/pods/1d070387-3f9b-4da3-b58b-adfb5511956a/volumes"
Sep 10 16:06:01.993193 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:01.993179    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36689670-76ce-4696-96d8-2e238c562ab3" path="/var/lib/kubelet/pods/36689670-76ce-4696-96d8-2e238c562ab3/volumes"
Sep 10 16:06:07.876507 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:07.876459    2572 patch_prober.go:28] interesting pod/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 16:06:07.876936 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:07.876535    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd" podUID="a5810430-84a6-4e80-a3c0-43767806dbf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 16:06:08.245837 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:08.245743    2572 scope.go:117] "RemoveContainer" containerID="d880c27fbf0642e4b5163933cd1d344d4a683b7ba198d3a516788c5ebe2811be"
Sep 10 16:06:08.259541 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:08.259515    2572 scope.go:117] "RemoveContainer" containerID="b389090ca99f30161f19abc5f361b6b17d7bda3dea5617cbe30fb15808f2c6a9"
Sep 10 16:06:08.943599 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:08.943461    2572 generic.go:356] "Generic (PLEG): container finished" podID="a5810430-84a6-4e80-a3c0-43767806dbf0" containerID="956c0e1eda1211d3fd03016ea27a12237d99b805d1a1272b715f582e73534d0d" exitCode=0
Sep 10 16:06:08.943599 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:08.943541    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd" event={"ID":"a5810430-84a6-4e80-a3c0-43767806dbf0","Type":"ContainerDied","Data":"956c0e1eda1211d3fd03016ea27a12237d99b805d1a1272b715f582e73534d0d"}
Sep 10 16:06:08.944044 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:08.943986    2572 scope.go:117] "RemoveContainer" containerID="956c0e1eda1211d3fd03016ea27a12237d99b805d1a1272b715f582e73534d0d"
Sep 10 16:06:08.945145 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:08.945119    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:06:08.945248 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:08.945185    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:06:08.945248 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:08.945214    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:06:09.951665 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:09.951629    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd" event={"ID":"a5810430-84a6-4e80-a3c0-43767806dbf0","Type":"ContainerStarted","Data":"220d61b37eb93b0e596e45ebd287a1009a45f7f40b70fbbfbcc55146cff54f05"}
Sep 10 16:06:10.299940 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:10.299909    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:06:10.300227 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:10.300203    2572 patch_prober.go:28] interesting pod/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 10 16:06:10.300276 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:10.300251    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd" podUID="a5810430-84a6-4e80-a3c0-43767806dbf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 10 16:06:11.300075 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:11.300033    2572 patch_prober.go:28] interesting pod/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 10 16:06:11.300592 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:11.300092    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd" podUID="a5810430-84a6-4e80-a3c0-43767806dbf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 10 16:06:12.299869 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:12.299828    2572 patch_prober.go:28] interesting pod/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 10 16:06:12.300071 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:12.299887    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd" podUID="a5810430-84a6-4e80-a3c0-43767806dbf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 10 16:06:12.868575 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:12.868540    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"]
Sep 10 16:06:12.868979 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:12.868959    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d070387-3f9b-4da3-b58b-adfb5511956a" containerName="main"
Sep 10 16:06:12.869024 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:12.868984    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d070387-3f9b-4da3-b58b-adfb5511956a" containerName="main"
Sep 10 16:06:12.869060 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:12.869037    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36689670-76ce-4696-96d8-2e238c562ab3" containerName="main"
Sep 10 16:06:12.869060 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:12.869048    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="36689670-76ce-4696-96d8-2e238c562ab3" containerName="main"
Sep 10 16:06:12.869172 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:12.869158    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="36689670-76ce-4696-96d8-2e238c562ab3" containerName="main"
Sep 10 16:06:12.869207 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:12.869177    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d070387-3f9b-4da3-b58b-adfb5511956a" containerName="main"
Sep 10 16:06:12.872285 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:12.872264    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:12.875737 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:12.875716    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-50ed73\"/\"default-dockercfg-5qjdv\""
Sep 10 16:06:12.875884 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:12.875713    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-50ed73\"/\"rate-test-model-50ed73-kserve-self-signed-certs\""
Sep 10 16:06:12.876184 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:12.876166    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-50ed73\"/\"openshift-service-ca.crt\""
Sep 10 16:06:12.876184 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:12.876176    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-50ed73\"/\"kube-root-ca.crt\""
Sep 10 16:06:12.884742 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:12.884716    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"]
Sep 10 16:06:13.006995 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.006959    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-home\") pod \"rate-test-model-50ed73-kserve-74b6974c47-c95g2\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:13.007209 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.007010    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac0f91d-3b85-4704-8db0-756ef0d41152-tls-certs\") pod \"rate-test-model-50ed73-kserve-74b6974c47-c95g2\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:13.007209 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.007108    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d72rd\" (UniqueName: \"kubernetes.io/projected/9ac0f91d-3b85-4704-8db0-756ef0d41152-kube-api-access-d72rd\") pod \"rate-test-model-50ed73-kserve-74b6974c47-c95g2\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:13.007209 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.007186    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-dshm\") pod \"rate-test-model-50ed73-kserve-74b6974c47-c95g2\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:13.007426 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.007242    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-model-cache\") pod \"rate-test-model-50ed73-kserve-74b6974c47-c95g2\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:13.007426 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.007277    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-tmp-dir\") pod \"rate-test-model-50ed73-kserve-74b6974c47-c95g2\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:13.108582 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.108539    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-dshm\") pod \"rate-test-model-50ed73-kserve-74b6974c47-c95g2\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:13.108876 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.108851    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-model-cache\") pod \"rate-test-model-50ed73-kserve-74b6974c47-c95g2\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:13.108970 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.108896    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-tmp-dir\") pod \"rate-test-model-50ed73-kserve-74b6974c47-c95g2\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:13.108970 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.108956    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-home\") pod \"rate-test-model-50ed73-kserve-74b6974c47-c95g2\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:13.109080 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.109018    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac0f91d-3b85-4704-8db0-756ef0d41152-tls-certs\") pod \"rate-test-model-50ed73-kserve-74b6974c47-c95g2\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:13.109137 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.109086    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d72rd\" (UniqueName: \"kubernetes.io/projected/9ac0f91d-3b85-4704-8db0-756ef0d41152-kube-api-access-d72rd\") pod \"rate-test-model-50ed73-kserve-74b6974c47-c95g2\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:13.109351 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.109324    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-model-cache\") pod \"rate-test-model-50ed73-kserve-74b6974c47-c95g2\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:13.109525 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.109410    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-tmp-dir\") pod \"rate-test-model-50ed73-kserve-74b6974c47-c95g2\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:13.109629 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.109594    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-home\") pod \"rate-test-model-50ed73-kserve-74b6974c47-c95g2\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:13.111876 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.111850    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-dshm\") pod \"rate-test-model-50ed73-kserve-74b6974c47-c95g2\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:13.112500 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.112473    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac0f91d-3b85-4704-8db0-756ef0d41152-tls-certs\") pod \"rate-test-model-50ed73-kserve-74b6974c47-c95g2\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:13.122022 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.121934    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d72rd\" (UniqueName: \"kubernetes.io/projected/9ac0f91d-3b85-4704-8db0-756ef0d41152-kube-api-access-d72rd\") pod \"rate-test-model-50ed73-kserve-74b6974c47-c95g2\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") " pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:13.182905 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.182864    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:13.300227 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.300192    2572 patch_prober.go:28] interesting pod/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 10 16:06:13.300414 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.300255    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd" podUID="a5810430-84a6-4e80-a3c0-43767806dbf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 10 16:06:13.497355 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.497321    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"]
Sep 10 16:06:13.498372 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:06:13.498339    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ac0f91d_3b85_4704_8db0_756ef0d41152.slice/crio-d9c1389a500eca2f2536b6336dbf8d600c2438028684a0775c57bd34deb94791 WatchSource:0}: Error finding container d9c1389a500eca2f2536b6336dbf8d600c2438028684a0775c57bd34deb94791: Status 404 returned error can't find the container with id d9c1389a500eca2f2536b6336dbf8d600c2438028684a0775c57bd34deb94791
Sep 10 16:06:13.980656 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.980624    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2" event={"ID":"9ac0f91d-3b85-4704-8db0-756ef0d41152","Type":"ContainerStarted","Data":"8420f924789ee60086842e13ace7bb6f232b30cd481e8f651d8ded4bed262855"}
Sep 10 16:06:13.980656 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:13.980665    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2" event={"ID":"9ac0f91d-3b85-4704-8db0-756ef0d41152","Type":"ContainerStarted","Data":"d9c1389a500eca2f2536b6336dbf8d600c2438028684a0775c57bd34deb94791"}
Sep 10 16:06:14.000796 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:14.000744    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2" podStartSLOduration=2.000727591 podStartE2EDuration="2.000727591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:06:13.998585498 +0000 UTC m=+1446.670995581" watchObservedRunningTime="2026-09-10 16:06:14.000727591 +0000 UTC m=+1446.673137674"
Sep 10 16:06:14.300408 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:14.300377    2572 patch_prober.go:28] interesting pod/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 10 16:06:14.300613 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:14.300441    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd" podUID="a5810430-84a6-4e80-a3c0-43767806dbf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 10 16:06:15.304046 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:15.304017    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:06:15.304492 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:15.304279    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:06:15.305073 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:15.305058    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:06:17.783032 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:17.782979    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"]
Sep 10 16:06:17.786947 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:17.786913    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:17.795989 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:17.795963    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-81d3ea-openshift-default-dockercfg-66drh\""
Sep 10 16:06:17.840522 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:17.840465    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"]
Sep 10 16:06:17.963259 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:17.963212    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-workload-socket\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:17.963448 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:17.963334    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2gkr\" (UniqueName: \"kubernetes.io/projected/662e7519-0835-4568-890f-b9d52754344a-kube-api-access-j2gkr\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:17.963448 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:17.963389    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-workload-certs\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:17.963550 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:17.963465    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/662e7519-0835-4568-890f-b9d52754344a-istiod-ca-cert\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:17.963550 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:17.963496    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/662e7519-0835-4568-890f-b9d52754344a-istio-token\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:17.963550 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:17.963525    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-credential-socket\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:17.963659 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:17.963570    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/662e7519-0835-4568-890f-b9d52754344a-istio-podinfo\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:17.963659 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:17.963633    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-istio-data\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:17.963728 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:17.963704    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-istio-envoy\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.064714 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.064659    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2gkr\" (UniqueName: \"kubernetes.io/projected/662e7519-0835-4568-890f-b9d52754344a-kube-api-access-j2gkr\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.064714 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.064721    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-workload-certs\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.064991 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.064764    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/662e7519-0835-4568-890f-b9d52754344a-istiod-ca-cert\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.064991 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.064784    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/662e7519-0835-4568-890f-b9d52754344a-istio-token\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.064991 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.064823    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-credential-socket\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.064991 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.064957    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/662e7519-0835-4568-890f-b9d52754344a-istio-podinfo\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.065198 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.065014    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-istio-data\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.065198 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.065059    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-istio-envoy\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.065198 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.065170    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-workload-socket\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.065437 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.065222    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-workload-certs\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.065437 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.065280    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-credential-socket\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.065556 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.065484    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-istio-data\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.065614 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.065561    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-workload-socket\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.065614 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.065589    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/662e7519-0835-4568-890f-b9d52754344a-istiod-ca-cert\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.067621 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.067593    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-istio-envoy\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.067964 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.067937    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/662e7519-0835-4568-890f-b9d52754344a-istio-podinfo\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.080243 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.080206    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2gkr\" (UniqueName: \"kubernetes.io/projected/662e7519-0835-4568-890f-b9d52754344a-kube-api-access-j2gkr\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.080425 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.080337    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/662e7519-0835-4568-890f-b9d52754344a-istio-token\") pod \"e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") " pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.098413 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.098376    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:18.384126 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.384093    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"]
Sep 10 16:06:18.386125 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:06:18.386091    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662e7519_0835_4568_890f_b9d52754344a.slice/crio-1de1942bca85fa0e428adfe2cbcf7e4ae3fde6bd1c2a0f6f5d9eb3f6670164c6 WatchSource:0}: Error finding container 1de1942bca85fa0e428adfe2cbcf7e4ae3fde6bd1c2a0f6f5d9eb3f6670164c6: Status 404 returned error can't find the container with id 1de1942bca85fa0e428adfe2cbcf7e4ae3fde6bd1c2a0f6f5d9eb3f6670164c6
Sep 10 16:06:18.388527 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.388496    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:06:18.388607 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.388563    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:06:18.388607 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:18.388591    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:06:19.011736 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:19.011691    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g" event={"ID":"662e7519-0835-4568-890f-b9d52754344a","Type":"ContainerStarted","Data":"724f27da86cc1695b23238aee59aad86e098e7af5e681efc2724a2a97fe01817"}
Sep 10 16:06:19.012133 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:19.011759    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g" event={"ID":"662e7519-0835-4568-890f-b9d52754344a","Type":"ContainerStarted","Data":"1de1942bca85fa0e428adfe2cbcf7e4ae3fde6bd1c2a0f6f5d9eb3f6670164c6"}
Sep 10 16:06:19.044120 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:19.044059    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g" podStartSLOduration=2.044039834 podStartE2EDuration="2.044039834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:06:19.04351023 +0000 UTC m=+1451.715920313" watchObservedRunningTime="2026-09-10 16:06:19.044039834 +0000 UTC m=+1451.716449918"
Sep 10 16:06:19.099279 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:19.099232    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:19.105032 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:19.105002    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:20.017831 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:20.017795    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:20.018874 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:20.018847    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:23.183947 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:23.183905    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:23.184428 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:23.183963    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:23.196697 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:23.196661    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:24.051513 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:24.051476    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:06:36.018418 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.018380    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"]
Sep 10 16:06:36.022524 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.022498    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.027465 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.027434    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-67df2b\"/\"openshift-service-ca.crt\""
Sep 10 16:06:36.027465 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.027456    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-67df2b\"/\"kube-root-ca.crt\""
Sep 10 16:06:36.027701 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.027569    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-67df2b\"/\"default-dockercfg-6bf8n\""
Sep 10 16:06:36.027701 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.027436    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-67df2b\"/\"rate-test-model-67df2b-kserve-self-signed-certs\""
Sep 10 16:06:36.039869 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.039836    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"]
Sep 10 16:06:36.161126 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.161081    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-dshm\") pod \"rate-test-model-67df2b-kserve-578dcb56cf-vpcn2\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.161126 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.161129    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-home\") pod \"rate-test-model-67df2b-kserve-578dcb56cf-vpcn2\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.161419 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.161151    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-model-cache\") pod \"rate-test-model-67df2b-kserve-578dcb56cf-vpcn2\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.161419 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.161262    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk7kx\" (UniqueName: \"kubernetes.io/projected/c5f629de-3e52-4617-a482-0ef80e5779ef-kube-api-access-pk7kx\") pod \"rate-test-model-67df2b-kserve-578dcb56cf-vpcn2\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.161419 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.161308    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-tmp-dir\") pod \"rate-test-model-67df2b-kserve-578dcb56cf-vpcn2\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.161419 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.161341    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f629de-3e52-4617-a482-0ef80e5779ef-tls-certs\") pod \"rate-test-model-67df2b-kserve-578dcb56cf-vpcn2\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.261912 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.261871    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-dshm\") pod \"rate-test-model-67df2b-kserve-578dcb56cf-vpcn2\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.261912 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.261917    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-home\") pod \"rate-test-model-67df2b-kserve-578dcb56cf-vpcn2\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.262203 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.261949    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-model-cache\") pod \"rate-test-model-67df2b-kserve-578dcb56cf-vpcn2\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.262203 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.262006    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk7kx\" (UniqueName: \"kubernetes.io/projected/c5f629de-3e52-4617-a482-0ef80e5779ef-kube-api-access-pk7kx\") pod \"rate-test-model-67df2b-kserve-578dcb56cf-vpcn2\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.262203 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.262052    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-tmp-dir\") pod \"rate-test-model-67df2b-kserve-578dcb56cf-vpcn2\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.262203 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.262084    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f629de-3e52-4617-a482-0ef80e5779ef-tls-certs\") pod \"rate-test-model-67df2b-kserve-578dcb56cf-vpcn2\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.262572 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.262495    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-model-cache\") pod \"rate-test-model-67df2b-kserve-578dcb56cf-vpcn2\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.262572 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.262536    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-tmp-dir\") pod \"rate-test-model-67df2b-kserve-578dcb56cf-vpcn2\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.262696 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.262669    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-home\") pod \"rate-test-model-67df2b-kserve-578dcb56cf-vpcn2\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.264943 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.264918    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-dshm\") pod \"rate-test-model-67df2b-kserve-578dcb56cf-vpcn2\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.265108 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.265091    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f629de-3e52-4617-a482-0ef80e5779ef-tls-certs\") pod \"rate-test-model-67df2b-kserve-578dcb56cf-vpcn2\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.271849 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.271783    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk7kx\" (UniqueName: \"kubernetes.io/projected/c5f629de-3e52-4617-a482-0ef80e5779ef-kube-api-access-pk7kx\") pod \"rate-test-model-67df2b-kserve-578dcb56cf-vpcn2\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") " pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.333092 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.333037    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:36.641697 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:36.641658    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"]
Sep 10 16:06:36.642761 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:06:36.642727    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5f629de_3e52_4617_a482_0ef80e5779ef.slice/crio-8c89dd264ca675c793cb009bd463af6ab56d2b3ad27d51f2189b88337df3513a WatchSource:0}: Error finding container 8c89dd264ca675c793cb009bd463af6ab56d2b3ad27d51f2189b88337df3513a: Status 404 returned error can't find the container with id 8c89dd264ca675c793cb009bd463af6ab56d2b3ad27d51f2189b88337df3513a
Sep 10 16:06:37.125957 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:37.125916    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2" event={"ID":"c5f629de-3e52-4617-a482-0ef80e5779ef","Type":"ContainerStarted","Data":"e060e57d38748d2bd6847a3cee086db1b198593a94800b54c122ebd942b4c6cf"}
Sep 10 16:06:37.125957 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:37.125968    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2" event={"ID":"c5f629de-3e52-4617-a482-0ef80e5779ef","Type":"ContainerStarted","Data":"8c89dd264ca675c793cb009bd463af6ab56d2b3ad27d51f2189b88337df3513a"}
Sep 10 16:06:37.151732 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:37.151664    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2" podStartSLOduration=2.151646603 podStartE2EDuration="2.151646603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:06:37.149050325 +0000 UTC m=+1469.821460409" watchObservedRunningTime="2026-09-10 16:06:37.151646603 +0000 UTC m=+1469.824056707"
Sep 10 16:06:39.142783 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:39.142744    2572 generic.go:356] "Generic (PLEG): container finished" podID="662e7519-0835-4568-890f-b9d52754344a" containerID="724f27da86cc1695b23238aee59aad86e098e7af5e681efc2724a2a97fe01817" exitCode=0
Sep 10 16:06:39.143223 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:39.142818    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g" event={"ID":"662e7519-0835-4568-890f-b9d52754344a","Type":"ContainerDied","Data":"724f27da86cc1695b23238aee59aad86e098e7af5e681efc2724a2a97fe01817"}
Sep 10 16:06:39.143269 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:39.143255    2572 scope.go:117] "RemoveContainer" containerID="724f27da86cc1695b23238aee59aad86e098e7af5e681efc2724a2a97fe01817"
Sep 10 16:06:39.144503 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:39.144477    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:06:39.144581 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:39.144544    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:06:39.144619 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:39.144594    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:06:40.152031 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:40.151997    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g" event={"ID":"662e7519-0835-4568-890f-b9d52754344a","Type":"ContainerStarted","Data":"27e571f1d29ef8229075805f90d8ea34b52216d9906cc3d000dbfb841c322b91"}
Sep 10 16:06:41.098903 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:41.098858    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:41.099125 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:41.099080    2572 patch_prober.go:28] interesting pod/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 10 16:06:41.099210 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:41.099138    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g" podUID="662e7519-0835-4568-890f-b9d52754344a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 10 16:06:42.098892 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:42.098848    2572 patch_prober.go:28] interesting pod/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 10 16:06:42.099522 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:42.098943    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g" podUID="662e7519-0835-4568-890f-b9d52754344a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 10 16:06:43.098942 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:43.098899    2572 patch_prober.go:28] interesting pod/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 10 16:06:43.099385 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:43.098963    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g" podUID="662e7519-0835-4568-890f-b9d52754344a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 10 16:06:44.099819 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:44.099778    2572 patch_prober.go:28] interesting pod/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 10 16:06:44.100353 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:44.099846    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g" podUID="662e7519-0835-4568-890f-b9d52754344a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 10 16:06:45.099020 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:45.098981    2572 patch_prober.go:28] interesting pod/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 10 16:06:45.099218 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:45.099042    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g" podUID="662e7519-0835-4568-890f-b9d52754344a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 10 16:06:46.103639 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:46.103606    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:46.104093 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:46.103880    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:46.104604 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:46.104587    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:06:46.334025 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:46.333977    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:46.334025 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:46.334032    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:46.346852 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:46.346814    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:47.212377 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:47.212345    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:06:48.864946 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:48.864898    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45e544-njw4p"]
Sep 10 16:06:48.867612 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:48.867592    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45e544-njw4p"
Sep 10 16:06:48.869933 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:48.869909    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:06:48.870486 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:48.870466    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\""
Sep 10 16:06:48.878659 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:48.878628    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45e544-njw4p"]
Sep 10 16:06:48.998475 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:48.998434    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8508b5a9-145d-4d25-ae76-d51d95b66cab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-45e544-njw4p\" (UID: \"8508b5a9-145d-4d25-ae76-d51d95b66cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45e544-njw4p"
Sep 10 16:06:49.099737 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:49.099695    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8508b5a9-145d-4d25-ae76-d51d95b66cab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-45e544-njw4p\" (UID: \"8508b5a9-145d-4d25-ae76-d51d95b66cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45e544-njw4p"
Sep 10 16:06:49.100353 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:49.100332    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8508b5a9-145d-4d25-ae76-d51d95b66cab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-45e544-njw4p\" (UID: \"8508b5a9-145d-4d25-ae76-d51d95b66cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45e544-njw4p"
Sep 10 16:06:49.180760 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:49.180665    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45e544-njw4p"
Sep 10 16:06:49.456608 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:49.456577    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45e544-njw4p"]
Sep 10 16:06:49.458619 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:06:49.458590    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8508b5a9_145d_4d25_ae76_d51d95b66cab.slice/crio-b2e7ab6dba730b03eb6a7b6066be33d7a356a04bdab67dca8daa1a9d9df7248b WatchSource:0}: Error finding container b2e7ab6dba730b03eb6a7b6066be33d7a356a04bdab67dca8daa1a9d9df7248b: Status 404 returned error can't find the container with id b2e7ab6dba730b03eb6a7b6066be33d7a356a04bdab67dca8daa1a9d9df7248b
Sep 10 16:06:50.221199 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:50.221158    2572 generic.go:356] "Generic (PLEG): container finished" podID="8508b5a9-145d-4d25-ae76-d51d95b66cab" containerID="b597a41a9cbb925c474dc78eb126b4753959512854774768653c5dfd747a762f" exitCode=0
Sep 10 16:06:50.221648 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:50.221243    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45e544-njw4p" event={"ID":"8508b5a9-145d-4d25-ae76-d51d95b66cab","Type":"ContainerDied","Data":"b597a41a9cbb925c474dc78eb126b4753959512854774768653c5dfd747a762f"}
Sep 10 16:06:50.221648 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:50.221325    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45e544-njw4p" event={"ID":"8508b5a9-145d-4d25-ae76-d51d95b66cab","Type":"ContainerStarted","Data":"b2e7ab6dba730b03eb6a7b6066be33d7a356a04bdab67dca8daa1a9d9df7248b"}
Sep 10 16:06:51.365594 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:51.365564    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45e544-njw4p"
Sep 10 16:06:51.526053 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:51.525940    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8508b5a9-145d-4d25-ae76-d51d95b66cab-maas-api-service-ca\") pod \"8508b5a9-145d-4d25-ae76-d51d95b66cab\" (UID: \"8508b5a9-145d-4d25-ae76-d51d95b66cab\") "
Sep 10 16:06:51.526408 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:51.526381    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8508b5a9-145d-4d25-ae76-d51d95b66cab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8508b5a9-145d-4d25-ae76-d51d95b66cab" (UID: "8508b5a9-145d-4d25-ae76-d51d95b66cab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:06:51.627679 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:51.627638    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8508b5a9-145d-4d25-ae76-d51d95b66cab-maas-api-service-ca\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:06:52.236384 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:52.236318    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45e544-njw4p"
Sep 10 16:06:52.236384 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:52.236335    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45e544-njw4p" event={"ID":"8508b5a9-145d-4d25-ae76-d51d95b66cab","Type":"ContainerDied","Data":"b2e7ab6dba730b03eb6a7b6066be33d7a356a04bdab67dca8daa1a9d9df7248b"}
Sep 10 16:06:52.236384 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:06:52.236383    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2e7ab6dba730b03eb6a7b6066be33d7a356a04bdab67dca8daa1a9d9df7248b"
Sep 10 16:07:05.024198 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:05.024153    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45e544-njw4p"]
Sep 10 16:07:05.027132 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:05.027104    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-45e544-njw4p"]
Sep 10 16:07:05.992937 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:05.992894    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8508b5a9-145d-4d25-ae76-d51d95b66cab" path="/var/lib/kubelet/pods/8508b5a9-145d-4d25-ae76-d51d95b66cab/volumes"
Sep 10 16:07:07.928485 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:07.928451    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7sm72_8ddd4dc1-1122-41e0-a54a-67fb8008d7b9/ovn-acl-logging/0.log"
Sep 10 16:07:07.934288 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:07.934262    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7sm72_8ddd4dc1-1122-41e0-a54a-67fb8008d7b9/ovn-acl-logging/0.log"
Sep 10 16:07:08.306158 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:08.306119    2572 scope.go:117] "RemoveContainer" containerID="afb940e28acd481d40caaded52455005cce86e2921b393eae9e08805787745e7"
Sep 10 16:07:11.850824 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:11.850785    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"]
Sep 10 16:07:11.851394 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:11.851046    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd" podUID="a5810430-84a6-4e80-a3c0-43767806dbf0" containerName="istio-proxy" containerID="cri-o://220d61b37eb93b0e596e45ebd287a1009a45f7f40b70fbbfbcc55146cff54f05" gracePeriod=30
Sep 10 16:07:12.215372 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:12.215271    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd"]
Sep 10 16:07:12.215710 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:12.215697    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8508b5a9-145d-4d25-ae76-d51d95b66cab" containerName="revoke-keys"
Sep 10 16:07:12.215759 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:12.215711    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8508b5a9-145d-4d25-ae76-d51d95b66cab" containerName="revoke-keys"
Sep 10 16:07:12.215803 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:12.215773    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8508b5a9-145d-4d25-ae76-d51d95b66cab" containerName="revoke-keys"
Sep 10 16:07:12.217951 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:12.217931    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd"
Sep 10 16:07:12.221232 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:12.221206    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\""
Sep 10 16:07:12.221624 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:12.221604    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:07:12.232410 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:12.232377    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd"]
Sep 10 16:07:12.337058 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:12.337018    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e647757d-d4f6-48aa-a891-0cadbea7d16b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd\" (UID: \"e647757d-d4f6-48aa-a891-0cadbea7d16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd"
Sep 10 16:07:12.437731 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:12.437686    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e647757d-d4f6-48aa-a891-0cadbea7d16b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd\" (UID: \"e647757d-d4f6-48aa-a891-0cadbea7d16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd"
Sep 10 16:07:12.438448 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:12.438422    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e647757d-d4f6-48aa-a891-0cadbea7d16b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd\" (UID: \"e647757d-d4f6-48aa-a891-0cadbea7d16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd"
Sep 10 16:07:12.529786 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:12.529676    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd"
Sep 10 16:07:12.804197 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:12.804164    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd"]
Sep 10 16:07:12.805249 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:07:12.805213    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode647757d_d4f6_48aa_a891_0cadbea7d16b.slice/crio-8f064c65e1623fa6c9443b0238506553f7b8ae3e81da0a1d5c9a1a4d53d0d33f WatchSource:0}: Error finding container 8f064c65e1623fa6c9443b0238506553f7b8ae3e81da0a1d5c9a1a4d53d0d33f: Status 404 returned error can't find the container with id 8f064c65e1623fa6c9443b0238506553f7b8ae3e81da0a1d5c9a1a4d53d0d33f
Sep 10 16:07:13.365041 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:13.365004    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd" event={"ID":"e647757d-d4f6-48aa-a891-0cadbea7d16b","Type":"ContainerStarted","Data":"aba58fb003cc8d5d42a5212dc3ab3da3a4b4f321c688e5b353194ffd0a8286c2"}
Sep 10 16:07:13.365575 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:13.365062    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd" event={"ID":"e647757d-d4f6-48aa-a891-0cadbea7d16b","Type":"ContainerStarted","Data":"8f064c65e1623fa6c9443b0238506553f7b8ae3e81da0a1d5c9a1a4d53d0d33f"}
Sep 10 16:07:13.400033 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:13.399918    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd" podStartSLOduration=1.399901167 podStartE2EDuration="1.399901167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:07:13.397386544 +0000 UTC m=+1506.069796628" watchObservedRunningTime="2026-09-10 16:07:13.399901167 +0000 UTC m=+1506.072311284"
Sep 10 16:07:14.372453 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:14.372415    2572 generic.go:356] "Generic (PLEG): container finished" podID="e647757d-d4f6-48aa-a891-0cadbea7d16b" containerID="aba58fb003cc8d5d42a5212dc3ab3da3a4b4f321c688e5b353194ffd0a8286c2" exitCode=0
Sep 10 16:07:14.372958 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:14.372507    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd" event={"ID":"e647757d-d4f6-48aa-a891-0cadbea7d16b","Type":"ContainerDied","Data":"aba58fb003cc8d5d42a5212dc3ab3da3a4b4f321c688e5b353194ffd0a8286c2"}
Sep 10 16:07:15.306670 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:15.306628    2572 patch_prober.go:28] interesting pod/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 16:07:15.306889 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:15.306701    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd" podUID="a5810430-84a6-4e80-a3c0-43767806dbf0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 16:07:15.515387 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:15.515344    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd"
Sep 10 16:07:15.670281 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:15.670189    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e647757d-d4f6-48aa-a891-0cadbea7d16b-maas-api-service-ca\") pod \"e647757d-d4f6-48aa-a891-0cadbea7d16b\" (UID: \"e647757d-d4f6-48aa-a891-0cadbea7d16b\") "
Sep 10 16:07:15.670596 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:15.670571    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e647757d-d4f6-48aa-a891-0cadbea7d16b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e647757d-d4f6-48aa-a891-0cadbea7d16b" (UID: "e647757d-d4f6-48aa-a891-0cadbea7d16b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:07:15.772069 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:15.772025    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e647757d-d4f6-48aa-a891-0cadbea7d16b-maas-api-service-ca\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:16.385025 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:16.384987    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd"
Sep 10 16:07:16.385231 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:16.385019    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd" event={"ID":"e647757d-d4f6-48aa-a891-0cadbea7d16b","Type":"ContainerDied","Data":"8f064c65e1623fa6c9443b0238506553f7b8ae3e81da0a1d5c9a1a4d53d0d33f"}
Sep 10 16:07:16.385231 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:16.385080    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f064c65e1623fa6c9443b0238506553f7b8ae3e81da0a1d5c9a1a4d53d0d33f"
Sep 10 16:07:17.110476 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.110446    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:07:17.185536 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.185441    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnt4w\" (UniqueName: \"kubernetes.io/projected/a5810430-84a6-4e80-a3c0-43767806dbf0-kube-api-access-rnt4w\") pod \"a5810430-84a6-4e80-a3c0-43767806dbf0\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") "
Sep 10 16:07:17.185536 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.185506    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5810430-84a6-4e80-a3c0-43767806dbf0-istiod-ca-cert\") pod \"a5810430-84a6-4e80-a3c0-43767806dbf0\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") "
Sep 10 16:07:17.185794 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.185553    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-data\") pod \"a5810430-84a6-4e80-a3c0-43767806dbf0\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") "
Sep 10 16:07:17.185794 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.185579    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-workload-socket\") pod \"a5810430-84a6-4e80-a3c0-43767806dbf0\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") "
Sep 10 16:07:17.185794 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.185609    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-podinfo\") pod \"a5810430-84a6-4e80-a3c0-43767806dbf0\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") "
Sep 10 16:07:17.185794 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.185675    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-credential-socket\") pod \"a5810430-84a6-4e80-a3c0-43767806dbf0\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") "
Sep 10 16:07:17.185794 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.185701    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-envoy\") pod \"a5810430-84a6-4e80-a3c0-43767806dbf0\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") "
Sep 10 16:07:17.185794 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.185749    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-workload-certs\") pod \"a5810430-84a6-4e80-a3c0-43767806dbf0\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") "
Sep 10 16:07:17.185794 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.185779    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-token\") pod \"a5810430-84a6-4e80-a3c0-43767806dbf0\" (UID: \"a5810430-84a6-4e80-a3c0-43767806dbf0\") "
Sep 10 16:07:17.186171 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.185879    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5810430-84a6-4e80-a3c0-43767806dbf0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a5810430-84a6-4e80-a3c0-43767806dbf0" (UID: "a5810430-84a6-4e80-a3c0-43767806dbf0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:07:17.186171 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.185945    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a5810430-84a6-4e80-a3c0-43767806dbf0" (UID: "a5810430-84a6-4e80-a3c0-43767806dbf0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:17.186171 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.185963    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-data" (OuterVolumeSpecName: "istio-data") pod "a5810430-84a6-4e80-a3c0-43767806dbf0" (UID: "a5810430-84a6-4e80-a3c0-43767806dbf0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:17.186372 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.186198    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a5810430-84a6-4e80-a3c0-43767806dbf0" (UID: "a5810430-84a6-4e80-a3c0-43767806dbf0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:17.186372 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.186221    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a5810430-84a6-4e80-a3c0-43767806dbf0" (UID: "a5810430-84a6-4e80-a3c0-43767806dbf0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:17.186372 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.186228    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5810430-84a6-4e80-a3c0-43767806dbf0-istiod-ca-cert\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:17.186372 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.186247    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-data\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:17.186372 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.186258    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-workload-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:17.188153 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.188124    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5810430-84a6-4e80-a3c0-43767806dbf0-kube-api-access-rnt4w" (OuterVolumeSpecName: "kube-api-access-rnt4w") pod "a5810430-84a6-4e80-a3c0-43767806dbf0" (UID: "a5810430-84a6-4e80-a3c0-43767806dbf0"). InnerVolumeSpecName "kube-api-access-rnt4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:07:17.188153 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.188144    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a5810430-84a6-4e80-a3c0-43767806dbf0" (UID: "a5810430-84a6-4e80-a3c0-43767806dbf0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:07:17.188425 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.188405    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a5810430-84a6-4e80-a3c0-43767806dbf0" (UID: "a5810430-84a6-4e80-a3c0-43767806dbf0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:17.188521 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.188505    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-token" (OuterVolumeSpecName: "istio-token") pod "a5810430-84a6-4e80-a3c0-43767806dbf0" (UID: "a5810430-84a6-4e80-a3c0-43767806dbf0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:07:17.287828 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.287784    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-envoy\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:17.287828 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.287821    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-workload-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:17.287828 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.287830    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-token\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:17.287828 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.287839    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnt4w\" (UniqueName: \"kubernetes.io/projected/a5810430-84a6-4e80-a3c0-43767806dbf0-kube-api-access-rnt4w\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:17.288122 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.287849    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5810430-84a6-4e80-a3c0-43767806dbf0-istio-podinfo\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:17.288122 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.287857    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5810430-84a6-4e80-a3c0-43767806dbf0-credential-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:17.337655 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.337613    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"]
Sep 10 16:07:17.393351 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.393291    2572 generic.go:356] "Generic (PLEG): container finished" podID="a5810430-84a6-4e80-a3c0-43767806dbf0" containerID="220d61b37eb93b0e596e45ebd287a1009a45f7f40b70fbbfbcc55146cff54f05" exitCode=0
Sep 10 16:07:17.393351 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.393331    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd" event={"ID":"a5810430-84a6-4e80-a3c0-43767806dbf0","Type":"ContainerDied","Data":"220d61b37eb93b0e596e45ebd287a1009a45f7f40b70fbbfbcc55146cff54f05"}
Sep 10 16:07:17.393351 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.393355    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"
Sep 10 16:07:17.393635 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.393422    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd" event={"ID":"a5810430-84a6-4e80-a3c0-43767806dbf0","Type":"ContainerDied","Data":"0bd580a815f9ca924566baab3c4bb64cf611723714c621bf1ad857be441a56c0"}
Sep 10 16:07:17.393635 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.393447    2572 scope.go:117] "RemoveContainer" containerID="220d61b37eb93b0e596e45ebd287a1009a45f7f40b70fbbfbcc55146cff54f05"
Sep 10 16:07:17.406911 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.406890    2572 scope.go:117] "RemoveContainer" containerID="956c0e1eda1211d3fd03016ea27a12237d99b805d1a1272b715f582e73534d0d"
Sep 10 16:07:17.417762 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.417732    2572 scope.go:117] "RemoveContainer" containerID="220d61b37eb93b0e596e45ebd287a1009a45f7f40b70fbbfbcc55146cff54f05"
Sep 10 16:07:17.418175 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:07:17.418147    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"220d61b37eb93b0e596e45ebd287a1009a45f7f40b70fbbfbcc55146cff54f05\": container with ID starting with 220d61b37eb93b0e596e45ebd287a1009a45f7f40b70fbbfbcc55146cff54f05 not found: ID does not exist" containerID="220d61b37eb93b0e596e45ebd287a1009a45f7f40b70fbbfbcc55146cff54f05"
Sep 10 16:07:17.418318 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.418186    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"220d61b37eb93b0e596e45ebd287a1009a45f7f40b70fbbfbcc55146cff54f05"} err="failed to get container status \"220d61b37eb93b0e596e45ebd287a1009a45f7f40b70fbbfbcc55146cff54f05\": rpc error: code = NotFound desc = could not find container \"220d61b37eb93b0e596e45ebd287a1009a45f7f40b70fbbfbcc55146cff54f05\": container with ID starting with 220d61b37eb93b0e596e45ebd287a1009a45f7f40b70fbbfbcc55146cff54f05 not found: ID does not exist"
Sep 10 16:07:17.418318 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.418216    2572 scope.go:117] "RemoveContainer" containerID="956c0e1eda1211d3fd03016ea27a12237d99b805d1a1272b715f582e73534d0d"
Sep 10 16:07:17.418577 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:07:17.418559    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"956c0e1eda1211d3fd03016ea27a12237d99b805d1a1272b715f582e73534d0d\": container with ID starting with 956c0e1eda1211d3fd03016ea27a12237d99b805d1a1272b715f582e73534d0d not found: ID does not exist" containerID="956c0e1eda1211d3fd03016ea27a12237d99b805d1a1272b715f582e73534d0d"
Sep 10 16:07:17.418648 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.418585    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956c0e1eda1211d3fd03016ea27a12237d99b805d1a1272b715f582e73534d0d"} err="failed to get container status \"956c0e1eda1211d3fd03016ea27a12237d99b805d1a1272b715f582e73534d0d\": rpc error: code = NotFound desc = could not find container \"956c0e1eda1211d3fd03016ea27a12237d99b805d1a1272b715f582e73534d0d\": container with ID starting with 956c0e1eda1211d3fd03016ea27a12237d99b805d1a1272b715f582e73534d0d not found: ID does not exist"
Sep 10 16:07:17.418729 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.418706    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"]
Sep 10 16:07:17.424082 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.424052    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-45e544-openshift-default-55fbbbccf4-hhbpd"]
Sep 10 16:07:17.994390 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:17.994340    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5810430-84a6-4e80-a3c0-43767806dbf0" path="/var/lib/kubelet/pods/a5810430-84a6-4e80-a3c0-43767806dbf0/volumes"
Sep 10 16:07:18.358061 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:18.358018    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"]
Sep 10 16:07:19.176764 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.176727    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"]
Sep 10 16:07:19.177188 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.177169    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5810430-84a6-4e80-a3c0-43767806dbf0" containerName="istio-proxy"
Sep 10 16:07:19.177314 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.177191    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5810430-84a6-4e80-a3c0-43767806dbf0" containerName="istio-proxy"
Sep 10 16:07:19.177314 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.177229    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e647757d-d4f6-48aa-a891-0cadbea7d16b" containerName="revoke-keys"
Sep 10 16:07:19.177314 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.177240    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e647757d-d4f6-48aa-a891-0cadbea7d16b" containerName="revoke-keys"
Sep 10 16:07:19.177486 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.177375    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5810430-84a6-4e80-a3c0-43767806dbf0" containerName="istio-proxy"
Sep 10 16:07:19.177486 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.177390    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5810430-84a6-4e80-a3c0-43767806dbf0" containerName="istio-proxy"
Sep 10 16:07:19.177486 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.177407    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e647757d-d4f6-48aa-a891-0cadbea7d16b" containerName="revoke-keys"
Sep 10 16:07:19.181691 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.181668    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.184720 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.184700    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c0ec41-openshift-default-dockercfg-dsh59\""
Sep 10 16:07:19.204573 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.204537    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"]
Sep 10 16:07:19.306145 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.306099    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-workload-socket\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.306409 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.306170    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-token\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.306409 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.306209    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-workload-certs\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.306409 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.306232    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-podinfo\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.306409 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.306258    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-data\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.306409 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.306355    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istiod-ca-cert\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.306409 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.306400    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-credential-socket\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.306722 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.306516    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56986\" (UniqueName: \"kubernetes.io/projected/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-kube-api-access-56986\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.306722 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.306572    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-envoy\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.407545 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.407506    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-workload-certs\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.407545 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.407547    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-podinfo\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.408113 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.407608    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-data\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.408113 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.407750    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istiod-ca-cert\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.408113 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.407794    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-credential-socket\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.408113 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.407884    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56986\" (UniqueName: \"kubernetes.io/projected/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-kube-api-access-56986\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.408113 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.407901    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-workload-certs\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.408113 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.407946    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-data\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.408113 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.407954    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-envoy\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.408113 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.408019    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-workload-socket\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.408113 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.408083    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-token\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.408706 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.408134    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-credential-socket\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.408706 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.408391    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-workload-socket\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.408706 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.408597    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istiod-ca-cert\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.410171 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.410150    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-envoy\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.410314 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.410285    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-podinfo\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.418749 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.418693    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-token\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.420955 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.420924    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56986\" (UniqueName: \"kubernetes.io/projected/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-kube-api-access-56986\") pod \"e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") " pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.493388 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.493252    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:19.778417 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.778291    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"]
Sep 10 16:07:19.781593 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:07:19.781566    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb966694f_6e28_4ac8_b6f3_bd18b9f1cd8e.slice/crio-3941c68c9c109ccac19ad9f8f4f71ba107017694a28c09a7ecadd9364d09a0b7 WatchSource:0}: Error finding container 3941c68c9c109ccac19ad9f8f4f71ba107017694a28c09a7ecadd9364d09a0b7: Status 404 returned error can't find the container with id 3941c68c9c109ccac19ad9f8f4f71ba107017694a28c09a7ecadd9364d09a0b7
Sep 10 16:07:19.783949 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.783918    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:07:19.784024 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.783986    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:07:19.784024 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:19.784014    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:07:20.414237 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:20.414192    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6" event={"ID":"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e","Type":"ContainerStarted","Data":"edf0499c5c3bda1b57ba0a03356d10b3cb5869450a0b4ce24628d82916e1097b"}
Sep 10 16:07:20.414721 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:20.414263    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6" event={"ID":"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e","Type":"ContainerStarted","Data":"3941c68c9c109ccac19ad9f8f4f71ba107017694a28c09a7ecadd9364d09a0b7"}
Sep 10 16:07:20.438826 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:20.438766    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6" podStartSLOduration=1.438745443 podStartE2EDuration="1.438745443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:07:20.435148146 +0000 UTC m=+1513.107558232" watchObservedRunningTime="2026-09-10 16:07:20.438745443 +0000 UTC m=+1513.111155527"
Sep 10 16:07:20.493598 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:20.493553    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:20.499286 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:20.499256    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:21.421150 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:21.421116    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:21.422140 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:21.422120    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:28.312774 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:28.312733    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd"]
Sep 10 16:07:28.318899 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:28.318866    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-81d3ea-7jzwd"]
Sep 10 16:07:29.992682 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:29.992600    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e647757d-d4f6-48aa-a891-0cadbea7d16b" path="/var/lib/kubelet/pods/e647757d-d4f6-48aa-a891-0cadbea7d16b/volumes"
Sep 10 16:07:32.357604 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.357538    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2" podUID="9ac0f91d-3b85-4704-8db0-756ef0d41152" containerName="main" containerID="cri-o://8420f924789ee60086842e13ace7bb6f232b30cd481e8f651d8ded4bed262855" gracePeriod=45
Sep 10 16:07:32.493296 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.493249    2572 generic.go:356] "Generic (PLEG): container finished" podID="9ac0f91d-3b85-4704-8db0-756ef0d41152" containerID="8420f924789ee60086842e13ace7bb6f232b30cd481e8f651d8ded4bed262855" exitCode=0
Sep 10 16:07:32.493470 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.493332    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2" event={"ID":"9ac0f91d-3b85-4704-8db0-756ef0d41152","Type":"ContainerDied","Data":"8420f924789ee60086842e13ace7bb6f232b30cd481e8f651d8ded4bed262855"}
Sep 10 16:07:32.617091 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.617016    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:07:32.634941 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.634904    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-model-cache\") pod \"9ac0f91d-3b85-4704-8db0-756ef0d41152\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") "
Sep 10 16:07:32.634941 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.634957    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-home\") pod \"9ac0f91d-3b85-4704-8db0-756ef0d41152\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") "
Sep 10 16:07:32.635232 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.634981    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-dshm\") pod \"9ac0f91d-3b85-4704-8db0-756ef0d41152\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") "
Sep 10 16:07:32.635232 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.635037    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d72rd\" (UniqueName: \"kubernetes.io/projected/9ac0f91d-3b85-4704-8db0-756ef0d41152-kube-api-access-d72rd\") pod \"9ac0f91d-3b85-4704-8db0-756ef0d41152\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") "
Sep 10 16:07:32.635232 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.635082    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac0f91d-3b85-4704-8db0-756ef0d41152-tls-certs\") pod \"9ac0f91d-3b85-4704-8db0-756ef0d41152\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") "
Sep 10 16:07:32.635232 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.635123    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-tmp-dir\") pod \"9ac0f91d-3b85-4704-8db0-756ef0d41152\" (UID: \"9ac0f91d-3b85-4704-8db0-756ef0d41152\") "
Sep 10 16:07:32.635509 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.635267    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ac0f91d-3b85-4704-8db0-756ef0d41152" (UID: "9ac0f91d-3b85-4704-8db0-756ef0d41152"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:32.635509 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.635282    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-home" (OuterVolumeSpecName: "home") pod "9ac0f91d-3b85-4704-8db0-756ef0d41152" (UID: "9ac0f91d-3b85-4704-8db0-756ef0d41152"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:32.636650 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.635692    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ac0f91d-3b85-4704-8db0-756ef0d41152" (UID: "9ac0f91d-3b85-4704-8db0-756ef0d41152"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:32.636990 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.636921    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-home\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:32.636990 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.636954    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-model-cache\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:32.638099 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.638052    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-dshm" (OuterVolumeSpecName: "dshm") pod "9ac0f91d-3b85-4704-8db0-756ef0d41152" (UID: "9ac0f91d-3b85-4704-8db0-756ef0d41152"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:32.638099 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.638066    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ac0f91d-3b85-4704-8db0-756ef0d41152-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ac0f91d-3b85-4704-8db0-756ef0d41152" (UID: "9ac0f91d-3b85-4704-8db0-756ef0d41152"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:07:32.638099 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.638079    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac0f91d-3b85-4704-8db0-756ef0d41152-kube-api-access-d72rd" (OuterVolumeSpecName: "kube-api-access-d72rd") pod "9ac0f91d-3b85-4704-8db0-756ef0d41152" (UID: "9ac0f91d-3b85-4704-8db0-756ef0d41152"). InnerVolumeSpecName "kube-api-access-d72rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:07:32.737721 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.737672    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-tmp-dir\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:32.737721 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.737709    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac0f91d-3b85-4704-8db0-756ef0d41152-dshm\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:32.737721 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.737720    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d72rd\" (UniqueName: \"kubernetes.io/projected/9ac0f91d-3b85-4704-8db0-756ef0d41152-kube-api-access-d72rd\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:32.737721 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:32.737731    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac0f91d-3b85-4704-8db0-756ef0d41152-tls-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:33.377905 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.377841    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2" podUID="c5f629de-3e52-4617-a482-0ef80e5779ef" containerName="main" containerID="cri-o://e060e57d38748d2bd6847a3cee086db1b198593a94800b54c122ebd942b4c6cf" gracePeriod=45
Sep 10 16:07:33.503754 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.503710    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"
Sep 10 16:07:33.503984 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.503719    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2" event={"ID":"9ac0f91d-3b85-4704-8db0-756ef0d41152","Type":"ContainerDied","Data":"d9c1389a500eca2f2536b6336dbf8d600c2438028684a0775c57bd34deb94791"}
Sep 10 16:07:33.503984 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.503849    2572 scope.go:117] "RemoveContainer" containerID="8420f924789ee60086842e13ace7bb6f232b30cd481e8f651d8ded4bed262855"
Sep 10 16:07:33.506383 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.506336    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2" event={"ID":"c5f629de-3e52-4617-a482-0ef80e5779ef","Type":"ContainerDied","Data":"e060e57d38748d2bd6847a3cee086db1b198593a94800b54c122ebd942b4c6cf"}
Sep 10 16:07:33.506540 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.506282    2572 generic.go:356] "Generic (PLEG): container finished" podID="c5f629de-3e52-4617-a482-0ef80e5779ef" containerID="e060e57d38748d2bd6847a3cee086db1b198593a94800b54c122ebd942b4c6cf" exitCode=0
Sep 10 16:07:33.533348 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.533274    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"]
Sep 10 16:07:33.537918 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.537882    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-50ed73/rate-test-model-50ed73-kserve-74b6974c47-c95g2"]
Sep 10 16:07:33.641679 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.641601    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:07:33.748683 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.748633    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f629de-3e52-4617-a482-0ef80e5779ef-tls-certs\") pod \"c5f629de-3e52-4617-a482-0ef80e5779ef\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") "
Sep 10 16:07:33.748683 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.748681    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-tmp-dir\") pod \"c5f629de-3e52-4617-a482-0ef80e5779ef\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") "
Sep 10 16:07:33.748957 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.748706    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-model-cache\") pod \"c5f629de-3e52-4617-a482-0ef80e5779ef\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") "
Sep 10 16:07:33.748957 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.748736    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pk7kx\" (UniqueName: \"kubernetes.io/projected/c5f629de-3e52-4617-a482-0ef80e5779ef-kube-api-access-pk7kx\") pod \"c5f629de-3e52-4617-a482-0ef80e5779ef\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") "
Sep 10 16:07:33.748957 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.748781    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-home\") pod \"c5f629de-3e52-4617-a482-0ef80e5779ef\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") "
Sep 10 16:07:33.748957 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.748839    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-dshm\") pod \"c5f629de-3e52-4617-a482-0ef80e5779ef\" (UID: \"c5f629de-3e52-4617-a482-0ef80e5779ef\") "
Sep 10 16:07:33.749163 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.748969    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c5f629de-3e52-4617-a482-0ef80e5779ef" (UID: "c5f629de-3e52-4617-a482-0ef80e5779ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:33.749163 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.749024    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "c5f629de-3e52-4617-a482-0ef80e5779ef" (UID: "c5f629de-3e52-4617-a482-0ef80e5779ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:33.749163 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.749132    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-home" (OuterVolumeSpecName: "home") pod "c5f629de-3e52-4617-a482-0ef80e5779ef" (UID: "c5f629de-3e52-4617-a482-0ef80e5779ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:33.749322 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.749270    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-tmp-dir\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:33.749373 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.749294    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-model-cache\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:33.749373 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.749341    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-home\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:33.751057 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.751034    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-dshm" (OuterVolumeSpecName: "dshm") pod "c5f629de-3e52-4617-a482-0ef80e5779ef" (UID: "c5f629de-3e52-4617-a482-0ef80e5779ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:33.751519 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.751493    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5f629de-3e52-4617-a482-0ef80e5779ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c5f629de-3e52-4617-a482-0ef80e5779ef" (UID: "c5f629de-3e52-4617-a482-0ef80e5779ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:07:33.751636 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.751533    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f629de-3e52-4617-a482-0ef80e5779ef-kube-api-access-pk7kx" (OuterVolumeSpecName: "kube-api-access-pk7kx") pod "c5f629de-3e52-4617-a482-0ef80e5779ef" (UID: "c5f629de-3e52-4617-a482-0ef80e5779ef"). InnerVolumeSpecName "kube-api-access-pk7kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:07:33.850456 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.850411    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f629de-3e52-4617-a482-0ef80e5779ef-dshm\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:33.850456 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.850448    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f629de-3e52-4617-a482-0ef80e5779ef-tls-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:33.850456 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.850463    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk7kx\" (UniqueName: \"kubernetes.io/projected/c5f629de-3e52-4617-a482-0ef80e5779ef-kube-api-access-pk7kx\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:33.994156 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:33.994032    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ac0f91d-3b85-4704-8db0-756ef0d41152" path="/var/lib/kubelet/pods/9ac0f91d-3b85-4704-8db0-756ef0d41152/volumes"
Sep 10 16:07:34.515286 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:34.515252    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"
Sep 10 16:07:34.515781 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:34.515251    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2" event={"ID":"c5f629de-3e52-4617-a482-0ef80e5779ef","Type":"ContainerDied","Data":"8c89dd264ca675c793cb009bd463af6ab56d2b3ad27d51f2189b88337df3513a"}
Sep 10 16:07:34.515781 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:34.515384    2572 scope.go:117] "RemoveContainer" containerID="e060e57d38748d2bd6847a3cee086db1b198593a94800b54c122ebd942b4c6cf"
Sep 10 16:07:34.539510 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:34.539474    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"]
Sep 10 16:07:34.543353 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:34.543318    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-67df2b/rate-test-model-67df2b-kserve-578dcb56cf-vpcn2"]
Sep 10 16:07:35.474945 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:35.474907    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"]
Sep 10 16:07:35.475256 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:35.475221    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g" podUID="662e7519-0835-4568-890f-b9d52754344a" containerName="istio-proxy" containerID="cri-o://27e571f1d29ef8229075805f90d8ea34b52216d9906cc3d000dbfb841c322b91" gracePeriod=30
Sep 10 16:07:35.854320 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:35.854271    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh"]
Sep 10 16:07:35.854770 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:35.854720    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ac0f91d-3b85-4704-8db0-756ef0d41152" containerName="main"
Sep 10 16:07:35.854770 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:35.854734    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ac0f91d-3b85-4704-8db0-756ef0d41152" containerName="main"
Sep 10 16:07:35.854770 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:35.854758    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5810430-84a6-4e80-a3c0-43767806dbf0" containerName="istio-proxy"
Sep 10 16:07:35.854770 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:35.854764    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5810430-84a6-4e80-a3c0-43767806dbf0" containerName="istio-proxy"
Sep 10 16:07:35.854955 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:35.854776    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5f629de-3e52-4617-a482-0ef80e5779ef" containerName="main"
Sep 10 16:07:35.854955 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:35.854783    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5f629de-3e52-4617-a482-0ef80e5779ef" containerName="main"
Sep 10 16:07:35.854955 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:35.854834    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ac0f91d-3b85-4704-8db0-756ef0d41152" containerName="main"
Sep 10 16:07:35.854955 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:35.854847    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5f629de-3e52-4617-a482-0ef80e5779ef" containerName="main"
Sep 10 16:07:35.859196 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:35.859176    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh"
Sep 10 16:07:35.861913 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:35.861886    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:07:35.862062 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:35.861977    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\""
Sep 10 16:07:35.869695 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:35.869663    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh"]
Sep 10 16:07:35.972751 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:35.972710    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42bfbb8-b637-4bd6-b557-eda0ff17f9cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh\" (UID: \"f42bfbb8-b637-4bd6-b557-eda0ff17f9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh"
Sep 10 16:07:35.992650 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:35.992615    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5f629de-3e52-4617-a482-0ef80e5779ef" path="/var/lib/kubelet/pods/c5f629de-3e52-4617-a482-0ef80e5779ef/volumes"
Sep 10 16:07:36.073631 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:36.073595    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42bfbb8-b637-4bd6-b557-eda0ff17f9cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh\" (UID: \"f42bfbb8-b637-4bd6-b557-eda0ff17f9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh"
Sep 10 16:07:36.074272 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:36.074253    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42bfbb8-b637-4bd6-b557-eda0ff17f9cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh\" (UID: \"f42bfbb8-b637-4bd6-b557-eda0ff17f9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh"
Sep 10 16:07:36.171572 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:36.171472    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh"
Sep 10 16:07:36.488745 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:36.488718    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh"]
Sep 10 16:07:36.491575 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:07:36.491542    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42bfbb8_b637_4bd6_b557_eda0ff17f9cf.slice/crio-91561f41035206e0d15bff4fee4ac5a9dfc87a6f9302b8bd452cbc1bda99c7bd WatchSource:0}: Error finding container 91561f41035206e0d15bff4fee4ac5a9dfc87a6f9302b8bd452cbc1bda99c7bd: Status 404 returned error can't find the container with id 91561f41035206e0d15bff4fee4ac5a9dfc87a6f9302b8bd452cbc1bda99c7bd
Sep 10 16:07:36.545839 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:36.545731    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh" event={"ID":"f42bfbb8-b637-4bd6-b557-eda0ff17f9cf","Type":"ContainerStarted","Data":"91561f41035206e0d15bff4fee4ac5a9dfc87a6f9302b8bd452cbc1bda99c7bd"}
Sep 10 16:07:37.554985 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:37.554949    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh" event={"ID":"f42bfbb8-b637-4bd6-b557-eda0ff17f9cf","Type":"ContainerStarted","Data":"fcc79c8f6c28ab650ee370246afaea054d256100a72d3f4e743883988ed0463b"}
Sep 10 16:07:37.577552 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:37.577492    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh" podStartSLOduration=2.577472502 podStartE2EDuration="2.577472502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:07:37.573881822 +0000 UTC m=+1530.246291904" watchObservedRunningTime="2026-09-10 16:07:37.577472502 +0000 UTC m=+1530.249882585"
Sep 10 16:07:38.563064 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:38.563021    2572 generic.go:356] "Generic (PLEG): container finished" podID="f42bfbb8-b637-4bd6-b557-eda0ff17f9cf" containerID="fcc79c8f6c28ab650ee370246afaea054d256100a72d3f4e743883988ed0463b" exitCode=0
Sep 10 16:07:38.563486 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:38.563105    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh" event={"ID":"f42bfbb8-b637-4bd6-b557-eda0ff17f9cf","Type":"ContainerDied","Data":"fcc79c8f6c28ab650ee370246afaea054d256100a72d3f4e743883988ed0463b"}
Sep 10 16:07:39.744603 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:39.744560    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh"
Sep 10 16:07:39.813571 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:39.813528    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42bfbb8-b637-4bd6-b557-eda0ff17f9cf-maas-api-service-ca\") pod \"f42bfbb8-b637-4bd6-b557-eda0ff17f9cf\" (UID: \"f42bfbb8-b637-4bd6-b557-eda0ff17f9cf\") "
Sep 10 16:07:39.813964 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:39.813936    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42bfbb8-b637-4bd6-b557-eda0ff17f9cf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f42bfbb8-b637-4bd6-b557-eda0ff17f9cf" (UID: "f42bfbb8-b637-4bd6-b557-eda0ff17f9cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:07:39.915399 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:39.915246    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42bfbb8-b637-4bd6-b557-eda0ff17f9cf-maas-api-service-ca\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:40.584693 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.584662    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh"
Sep 10 16:07:40.584903 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.584700    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh" event={"ID":"f42bfbb8-b637-4bd6-b557-eda0ff17f9cf","Type":"ContainerDied","Data":"91561f41035206e0d15bff4fee4ac5a9dfc87a6f9302b8bd452cbc1bda99c7bd"}
Sep 10 16:07:40.584982 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.584919    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91561f41035206e0d15bff4fee4ac5a9dfc87a6f9302b8bd452cbc1bda99c7bd"
Sep 10 16:07:40.780901 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.780869    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:07:40.837741 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.837702    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/662e7519-0835-4568-890f-b9d52754344a-istiod-ca-cert\") pod \"662e7519-0835-4568-890f-b9d52754344a\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") "
Sep 10 16:07:40.837951 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.837799    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2gkr\" (UniqueName: \"kubernetes.io/projected/662e7519-0835-4568-890f-b9d52754344a-kube-api-access-j2gkr\") pod \"662e7519-0835-4568-890f-b9d52754344a\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") "
Sep 10 16:07:40.837951 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.837831    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/662e7519-0835-4568-890f-b9d52754344a-istio-token\") pod \"662e7519-0835-4568-890f-b9d52754344a\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") "
Sep 10 16:07:40.837951 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.837858    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-istio-data\") pod \"662e7519-0835-4568-890f-b9d52754344a\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") "
Sep 10 16:07:40.837951 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.837893    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-istio-envoy\") pod \"662e7519-0835-4568-890f-b9d52754344a\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") "
Sep 10 16:07:40.837951 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.837938    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-workload-certs\") pod \"662e7519-0835-4568-890f-b9d52754344a\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") "
Sep 10 16:07:40.838230 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.837961    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-credential-socket\") pod \"662e7519-0835-4568-890f-b9d52754344a\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") "
Sep 10 16:07:40.838230 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.838020    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-workload-socket\") pod \"662e7519-0835-4568-890f-b9d52754344a\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") "
Sep 10 16:07:40.838230 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.838044    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/662e7519-0835-4568-890f-b9d52754344a-istio-podinfo\") pod \"662e7519-0835-4568-890f-b9d52754344a\" (UID: \"662e7519-0835-4568-890f-b9d52754344a\") "
Sep 10 16:07:40.838230 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.838195    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662e7519-0835-4568-890f-b9d52754344a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "662e7519-0835-4568-890f-b9d52754344a" (UID: "662e7519-0835-4568-890f-b9d52754344a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:07:40.838478 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.838260    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "662e7519-0835-4568-890f-b9d52754344a" (UID: "662e7519-0835-4568-890f-b9d52754344a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:40.838533 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.838485    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-workload-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:40.838533 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.838508    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/662e7519-0835-4568-890f-b9d52754344a-istiod-ca-cert\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:40.838724 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.838691    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "662e7519-0835-4568-890f-b9d52754344a" (UID: "662e7519-0835-4568-890f-b9d52754344a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:40.838806 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.838715    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "662e7519-0835-4568-890f-b9d52754344a" (UID: "662e7519-0835-4568-890f-b9d52754344a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:40.839005 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.838979    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-istio-data" (OuterVolumeSpecName: "istio-data") pod "662e7519-0835-4568-890f-b9d52754344a" (UID: "662e7519-0835-4568-890f-b9d52754344a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:40.840508 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.840480    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/662e7519-0835-4568-890f-b9d52754344a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "662e7519-0835-4568-890f-b9d52754344a" (UID: "662e7519-0835-4568-890f-b9d52754344a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:07:40.840842 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.840820    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662e7519-0835-4568-890f-b9d52754344a-istio-token" (OuterVolumeSpecName: "istio-token") pod "662e7519-0835-4568-890f-b9d52754344a" (UID: "662e7519-0835-4568-890f-b9d52754344a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:07:40.841054 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.841036    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662e7519-0835-4568-890f-b9d52754344a-kube-api-access-j2gkr" (OuterVolumeSpecName: "kube-api-access-j2gkr") pod "662e7519-0835-4568-890f-b9d52754344a" (UID: "662e7519-0835-4568-890f-b9d52754344a"). InnerVolumeSpecName "kube-api-access-j2gkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:07:40.841137 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.841066    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "662e7519-0835-4568-890f-b9d52754344a" (UID: "662e7519-0835-4568-890f-b9d52754344a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:40.939165 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.939127    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2gkr\" (UniqueName: \"kubernetes.io/projected/662e7519-0835-4568-890f-b9d52754344a-kube-api-access-j2gkr\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:40.939165 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.939163    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/662e7519-0835-4568-890f-b9d52754344a-istio-token\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:40.939388 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.939175    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-istio-data\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:40.939388 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.939186    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-istio-envoy\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:40.939388 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.939198    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-credential-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:40.939388 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.939209    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/662e7519-0835-4568-890f-b9d52754344a-workload-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:40.939388 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:40.939219    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/662e7519-0835-4568-890f-b9d52754344a-istio-podinfo\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:41.593441 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:41.593401    2572 generic.go:356] "Generic (PLEG): container finished" podID="662e7519-0835-4568-890f-b9d52754344a" containerID="27e571f1d29ef8229075805f90d8ea34b52216d9906cc3d000dbfb841c322b91" exitCode=0
Sep 10 16:07:41.593601 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:41.593460    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"
Sep 10 16:07:41.593601 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:41.593489    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g" event={"ID":"662e7519-0835-4568-890f-b9d52754344a","Type":"ContainerDied","Data":"27e571f1d29ef8229075805f90d8ea34b52216d9906cc3d000dbfb841c322b91"}
Sep 10 16:07:41.593601 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:41.593551    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g" event={"ID":"662e7519-0835-4568-890f-b9d52754344a","Type":"ContainerDied","Data":"1de1942bca85fa0e428adfe2cbcf7e4ae3fde6bd1c2a0f6f5d9eb3f6670164c6"}
Sep 10 16:07:41.593601 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:41.593582    2572 scope.go:117] "RemoveContainer" containerID="27e571f1d29ef8229075805f90d8ea34b52216d9906cc3d000dbfb841c322b91"
Sep 10 16:07:41.604058 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:41.603883    2572 scope.go:117] "RemoveContainer" containerID="724f27da86cc1695b23238aee59aad86e098e7af5e681efc2724a2a97fe01817"
Sep 10 16:07:41.613184 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:41.613160    2572 scope.go:117] "RemoveContainer" containerID="27e571f1d29ef8229075805f90d8ea34b52216d9906cc3d000dbfb841c322b91"
Sep 10 16:07:41.613540 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:07:41.613518    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27e571f1d29ef8229075805f90d8ea34b52216d9906cc3d000dbfb841c322b91\": container with ID starting with 27e571f1d29ef8229075805f90d8ea34b52216d9906cc3d000dbfb841c322b91 not found: ID does not exist" containerID="27e571f1d29ef8229075805f90d8ea34b52216d9906cc3d000dbfb841c322b91"
Sep 10 16:07:41.613621 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:41.613550    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e571f1d29ef8229075805f90d8ea34b52216d9906cc3d000dbfb841c322b91"} err="failed to get container status \"27e571f1d29ef8229075805f90d8ea34b52216d9906cc3d000dbfb841c322b91\": rpc error: code = NotFound desc = could not find container \"27e571f1d29ef8229075805f90d8ea34b52216d9906cc3d000dbfb841c322b91\": container with ID starting with 27e571f1d29ef8229075805f90d8ea34b52216d9906cc3d000dbfb841c322b91 not found: ID does not exist"
Sep 10 16:07:41.613621 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:41.613570    2572 scope.go:117] "RemoveContainer" containerID="724f27da86cc1695b23238aee59aad86e098e7af5e681efc2724a2a97fe01817"
Sep 10 16:07:41.613848 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:07:41.613829    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724f27da86cc1695b23238aee59aad86e098e7af5e681efc2724a2a97fe01817\": container with ID starting with 724f27da86cc1695b23238aee59aad86e098e7af5e681efc2724a2a97fe01817 not found: ID does not exist" containerID="724f27da86cc1695b23238aee59aad86e098e7af5e681efc2724a2a97fe01817"
Sep 10 16:07:41.613907 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:41.613861    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724f27da86cc1695b23238aee59aad86e098e7af5e681efc2724a2a97fe01817"} err="failed to get container status \"724f27da86cc1695b23238aee59aad86e098e7af5e681efc2724a2a97fe01817\": rpc error: code = NotFound desc = could not find container \"724f27da86cc1695b23238aee59aad86e098e7af5e681efc2724a2a97fe01817\": container with ID starting with 724f27da86cc1695b23238aee59aad86e098e7af5e681efc2724a2a97fe01817 not found: ID does not exist"
Sep 10 16:07:41.619902 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:41.619866    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"]
Sep 10 16:07:41.623699 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:41.623666    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-81d3ea-openshift-default-644ff8f66b-sfk9g"]
Sep 10 16:07:41.993700 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:41.993610    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="662e7519-0835-4568-890f-b9d52754344a" path="/var/lib/kubelet/pods/662e7519-0835-4568-890f-b9d52754344a/volumes"
Sep 10 16:07:44.615156 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:44.615124    2572 generic.go:356] "Generic (PLEG): container finished" podID="b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" containerID="edf0499c5c3bda1b57ba0a03356d10b3cb5869450a0b4ce24628d82916e1097b" exitCode=0
Sep 10 16:07:44.615730 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:44.615179    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6" event={"ID":"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e","Type":"ContainerDied","Data":"edf0499c5c3bda1b57ba0a03356d10b3cb5869450a0b4ce24628d82916e1097b"}
Sep 10 16:07:44.615730 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:44.615594    2572 scope.go:117] "RemoveContainer" containerID="edf0499c5c3bda1b57ba0a03356d10b3cb5869450a0b4ce24628d82916e1097b"
Sep 10 16:07:44.617079 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:44.617045    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:07:44.617183 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:44.617117    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:07:44.617183 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:44.617148    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 16:07:45.626291 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:45.626238    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6" event={"ID":"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e","Type":"ContainerStarted","Data":"928aa9ce66c336810d8b0ad18a8aa3b585e12233e67eb306872dc5a9c47e5a8c"}
Sep 10 16:07:46.425206 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:46.425163    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh"]
Sep 10 16:07:46.429198 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:46.429163    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a361a9-nmnxh"]
Sep 10 16:07:46.494476 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:46.494434    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:46.495566 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:46.495538    2572 patch_prober.go:28] interesting pod/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Sep 10 16:07:46.495693 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:46.495591    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6" podUID="b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 10 16:07:47.494206 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:47.494164    2572 patch_prober.go:28] interesting pod/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Sep 10 16:07:47.494648 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:47.494226    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6" podUID="b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 10 16:07:47.993173 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:47.993118    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f42bfbb8-b637-4bd6-b557-eda0ff17f9cf" path="/var/lib/kubelet/pods/f42bfbb8-b637-4bd6-b557-eda0ff17f9cf/volumes"
Sep 10 16:07:48.493909 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:48.493853    2572 patch_prober.go:28] interesting pod/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Sep 10 16:07:48.494136 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:48.493942    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6" podUID="b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 10 16:07:49.494249 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:49.494206    2572 patch_prober.go:28] interesting pod/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Sep 10 16:07:49.494732 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:49.494267    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6" podUID="b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 10 16:07:50.494676 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:50.494636    2572 patch_prober.go:28] interesting pod/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Sep 10 16:07:50.495090 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:50.494696    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6" podUID="b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 10 16:07:51.422039 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:51.421980    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:51.498792 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:51.498756    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:51.671393 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:51.671362    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:07:55.354752 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:55.354703    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg"]
Sep 10 16:07:55.355275 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:55.355252    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f42bfbb8-b637-4bd6-b557-eda0ff17f9cf" containerName="revoke-keys"
Sep 10 16:07:55.355404 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:55.355279    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f42bfbb8-b637-4bd6-b557-eda0ff17f9cf" containerName="revoke-keys"
Sep 10 16:07:55.355404 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:55.355341    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="662e7519-0835-4568-890f-b9d52754344a" containerName="istio-proxy"
Sep 10 16:07:55.355404 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:55.355350    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="662e7519-0835-4568-890f-b9d52754344a" containerName="istio-proxy"
Sep 10 16:07:55.355568 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:55.355451    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="662e7519-0835-4568-890f-b9d52754344a" containerName="istio-proxy"
Sep 10 16:07:55.355568 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:55.355465    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f42bfbb8-b637-4bd6-b557-eda0ff17f9cf" containerName="revoke-keys"
Sep 10 16:07:55.358183 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:55.358160    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg"
Sep 10 16:07:55.360519 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:55.360496    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:07:55.360647 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:55.360586    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\""
Sep 10 16:07:55.368873 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:55.368844    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg"]
Sep 10 16:07:55.494037 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:55.493992    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b25cf99-605e-40b4-a943-86e18c34820b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg\" (UID: \"3b25cf99-605e-40b4-a943-86e18c34820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg"
Sep 10 16:07:55.594605 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:55.594560    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b25cf99-605e-40b4-a943-86e18c34820b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg\" (UID: \"3b25cf99-605e-40b4-a943-86e18c34820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg"
Sep 10 16:07:55.595292 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:55.595264    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b25cf99-605e-40b4-a943-86e18c34820b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg\" (UID: \"3b25cf99-605e-40b4-a943-86e18c34820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg"
Sep 10 16:07:55.669997 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:55.669903    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg"
Sep 10 16:07:55.949434 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:55.949407    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg"]
Sep 10 16:07:55.952336 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:07:55.952284    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b25cf99_605e_40b4_a943_86e18c34820b.slice/crio-aa327d4736584786df508faf6479ab44d89fec3024b0a8203b57c701c4724724 WatchSource:0}: Error finding container aa327d4736584786df508faf6479ab44d89fec3024b0a8203b57c701c4724724: Status 404 returned error can't find the container with id aa327d4736584786df508faf6479ab44d89fec3024b0a8203b57c701c4724724
Sep 10 16:07:56.703211 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:56.703110    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg" event={"ID":"3b25cf99-605e-40b4-a943-86e18c34820b","Type":"ContainerStarted","Data":"8a7d8f4754847aef62651e6ba59f51f439049d89a186d1d6d3ff191024563659"}
Sep 10 16:07:56.703211 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:56.703168    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg" event={"ID":"3b25cf99-605e-40b4-a943-86e18c34820b","Type":"ContainerStarted","Data":"aa327d4736584786df508faf6479ab44d89fec3024b0a8203b57c701c4724724"}
Sep 10 16:07:56.722849 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:56.722781    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg" podStartSLOduration=1.722759148 podStartE2EDuration="1.722759148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:07:56.720018393 +0000 UTC m=+1549.392428475" watchObservedRunningTime="2026-09-10 16:07:56.722759148 +0000 UTC m=+1549.395169232"
Sep 10 16:07:57.710497 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:57.710457    2572 generic.go:356] "Generic (PLEG): container finished" podID="3b25cf99-605e-40b4-a943-86e18c34820b" containerID="8a7d8f4754847aef62651e6ba59f51f439049d89a186d1d6d3ff191024563659" exitCode=0
Sep 10 16:07:57.710899 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:57.710518    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg" event={"ID":"3b25cf99-605e-40b4-a943-86e18c34820b","Type":"ContainerDied","Data":"8a7d8f4754847aef62651e6ba59f51f439049d89a186d1d6d3ff191024563659"}
Sep 10 16:07:58.846648 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:58.846621    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg"
Sep 10 16:07:59.030000 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:59.029895    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b25cf99-605e-40b4-a943-86e18c34820b-maas-api-service-ca\") pod \"3b25cf99-605e-40b4-a943-86e18c34820b\" (UID: \"3b25cf99-605e-40b4-a943-86e18c34820b\") "
Sep 10 16:07:59.030351 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:59.030320    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b25cf99-605e-40b4-a943-86e18c34820b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b25cf99-605e-40b4-a943-86e18c34820b" (UID: "3b25cf99-605e-40b4-a943-86e18c34820b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:07:59.130685 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:59.130645    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b25cf99-605e-40b4-a943-86e18c34820b-maas-api-service-ca\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:07:59.723690 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:59.723656    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg" event={"ID":"3b25cf99-605e-40b4-a943-86e18c34820b","Type":"ContainerDied","Data":"aa327d4736584786df508faf6479ab44d89fec3024b0a8203b57c701c4724724"}
Sep 10 16:07:59.723891 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:59.723705    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa327d4736584786df508faf6479ab44d89fec3024b0a8203b57c701c4724724"
Sep 10 16:07:59.723891 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:07:59.723709    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg"
Sep 10 16:08:06.224677 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:06.224635    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg"]
Sep 10 16:08:06.227629 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:06.227602    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-33451c-pc6bg"]
Sep 10 16:08:07.992249 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:07.992202    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b25cf99-605e-40b4-a943-86e18c34820b" path="/var/lib/kubelet/pods/3b25cf99-605e-40b4-a943-86e18c34820b/volumes"
Sep 10 16:08:13.753707 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:13.753663    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"]
Sep 10 16:08:13.754144 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:13.753937    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n" podUID="2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" containerName="istio-proxy" containerID="cri-o://c5d8c0b34cdf240afc4ce3aa97c217ac5387943e566a942f919e0c3e632a40ba" gracePeriod=30
Sep 10 16:08:19.008710 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.008677    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:08:19.124930 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.124889    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-workload-socket\") pod \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") "
Sep 10 16:08:19.125140 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.124963    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-credential-socket\") pod \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") "
Sep 10 16:08:19.125140 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.125003    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxhxd\" (UniqueName: \"kubernetes.io/projected/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-kube-api-access-wxhxd\") pod \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") "
Sep 10 16:08:19.125140 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.125087    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istiod-ca-cert\") pod \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") "
Sep 10 16:08:19.125339 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.125155    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-token\") pod \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") "
Sep 10 16:08:19.125339 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.125186    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-workload-certs\") pod \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") "
Sep 10 16:08:19.125339 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.125222    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-podinfo\") pod \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") "
Sep 10 16:08:19.125339 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.125251    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" (UID: "2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:08:19.125339 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.125268    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-data\") pod \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") "
Sep 10 16:08:19.125339 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.125341    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-envoy\") pod \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\" (UID: \"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9\") "
Sep 10 16:08:19.125674 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.125374    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" (UID: "2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:08:19.125674 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.125471    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" (UID: "2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:08:19.125674 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.125523    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" (UID: "2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:08:19.125674 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.125650    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-data" (OuterVolumeSpecName: "istio-data") pod "2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" (UID: "2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:08:19.126124 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.126021    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-workload-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:08:19.126124 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.126048    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-data\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:08:19.126124 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.126061    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-workload-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:08:19.126124 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.126075    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-credential-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:08:19.126124 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.126090    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istiod-ca-cert\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:08:19.127960 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.127926    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-kube-api-access-wxhxd" (OuterVolumeSpecName: "kube-api-access-wxhxd") pod "2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" (UID: "2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9"). InnerVolumeSpecName "kube-api-access-wxhxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:08:19.127960 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.127925    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-token" (OuterVolumeSpecName: "istio-token") pod "2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" (UID: "2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:08:19.128107 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.128003    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" (UID: "2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:08:19.128107 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.128007    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" (UID: "2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:08:19.226764 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.226722    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-token\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:08:19.226764 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.226757    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-podinfo\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:08:19.226764 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.226767    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-istio-envoy\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:08:19.227006 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.226775    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxhxd\" (UniqueName: \"kubernetes.io/projected/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9-kube-api-access-wxhxd\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:08:19.841273 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.841238    2572 generic.go:356] "Generic (PLEG): container finished" podID="2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" containerID="c5d8c0b34cdf240afc4ce3aa97c217ac5387943e566a942f919e0c3e632a40ba" exitCode=0
Sep 10 16:08:19.841468 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.841292    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"
Sep 10 16:08:19.841468 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.841321    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n" event={"ID":"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9","Type":"ContainerDied","Data":"c5d8c0b34cdf240afc4ce3aa97c217ac5387943e566a942f919e0c3e632a40ba"}
Sep 10 16:08:19.841468 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.841369    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n" event={"ID":"2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9","Type":"ContainerDied","Data":"c21967c727aae3627448f63ae68db2bd6d98d465b85913778392f7ba221cea6a"}
Sep 10 16:08:19.841468 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.841391    2572 scope.go:117] "RemoveContainer" containerID="c5d8c0b34cdf240afc4ce3aa97c217ac5387943e566a942f919e0c3e632a40ba"
Sep 10 16:08:19.852596 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.852560    2572 scope.go:117] "RemoveContainer" containerID="ed6852e1fd45ed592a4240a4e8ff1dda5aad7f8c7f0874192680774ee0c3aca6"
Sep 10 16:08:19.862111 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.862089    2572 scope.go:117] "RemoveContainer" containerID="c5d8c0b34cdf240afc4ce3aa97c217ac5387943e566a942f919e0c3e632a40ba"
Sep 10 16:08:19.862518 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:08:19.862484    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5d8c0b34cdf240afc4ce3aa97c217ac5387943e566a942f919e0c3e632a40ba\": container with ID starting with c5d8c0b34cdf240afc4ce3aa97c217ac5387943e566a942f919e0c3e632a40ba not found: ID does not exist" containerID="c5d8c0b34cdf240afc4ce3aa97c217ac5387943e566a942f919e0c3e632a40ba"
Sep 10 16:08:19.862587 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.862521    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5d8c0b34cdf240afc4ce3aa97c217ac5387943e566a942f919e0c3e632a40ba"} err="failed to get container status \"c5d8c0b34cdf240afc4ce3aa97c217ac5387943e566a942f919e0c3e632a40ba\": rpc error: code = NotFound desc = could not find container \"c5d8c0b34cdf240afc4ce3aa97c217ac5387943e566a942f919e0c3e632a40ba\": container with ID starting with c5d8c0b34cdf240afc4ce3aa97c217ac5387943e566a942f919e0c3e632a40ba not found: ID does not exist"
Sep 10 16:08:19.862587 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.862539    2572 scope.go:117] "RemoveContainer" containerID="ed6852e1fd45ed592a4240a4e8ff1dda5aad7f8c7f0874192680774ee0c3aca6"
Sep 10 16:08:19.862880 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:08:19.862855    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed6852e1fd45ed592a4240a4e8ff1dda5aad7f8c7f0874192680774ee0c3aca6\": container with ID starting with ed6852e1fd45ed592a4240a4e8ff1dda5aad7f8c7f0874192680774ee0c3aca6 not found: ID does not exist" containerID="ed6852e1fd45ed592a4240a4e8ff1dda5aad7f8c7f0874192680774ee0c3aca6"
Sep 10 16:08:19.862991 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.862888    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed6852e1fd45ed592a4240a4e8ff1dda5aad7f8c7f0874192680774ee0c3aca6"} err="failed to get container status \"ed6852e1fd45ed592a4240a4e8ff1dda5aad7f8c7f0874192680774ee0c3aca6\": rpc error: code = NotFound desc = could not find container \"ed6852e1fd45ed592a4240a4e8ff1dda5aad7f8c7f0874192680774ee0c3aca6\": container with ID starting with ed6852e1fd45ed592a4240a4e8ff1dda5aad7f8c7f0874192680774ee0c3aca6 not found: ID does not exist"
Sep 10 16:08:19.876767 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.876700    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"]
Sep 10 16:08:19.879399 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.879365    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-33451c-openshift-default-59489f6894-gxw7n"]
Sep 10 16:08:19.992498 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:19.992463    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" path="/var/lib/kubelet/pods/2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9/volumes"
Sep 10 16:08:33.173362 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.173319    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"]
Sep 10 16:08:33.173942 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.173919    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" containerName="istio-proxy"
Sep 10 16:08:33.174007 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.173946    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" containerName="istio-proxy"
Sep 10 16:08:33.174007 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.173960    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b25cf99-605e-40b4-a943-86e18c34820b" containerName="revoke-keys"
Sep 10 16:08:33.174007 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.173969    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b25cf99-605e-40b4-a943-86e18c34820b" containerName="revoke-keys"
Sep 10 16:08:33.174007 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.173998    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="662e7519-0835-4568-890f-b9d52754344a" containerName="istio-proxy"
Sep 10 16:08:33.174007 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.174006    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="662e7519-0835-4568-890f-b9d52754344a" containerName="istio-proxy"
Sep 10 16:08:33.174170 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.174121    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" containerName="istio-proxy"
Sep 10 16:08:33.174170 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.174132    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b25cf99-605e-40b4-a943-86e18c34820b" containerName="revoke-keys"
Sep 10 16:08:33.174170 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.174141    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" containerName="istio-proxy"
Sep 10 16:08:33.174170 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.174149    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="662e7519-0835-4568-890f-b9d52754344a" containerName="istio-proxy"
Sep 10 16:08:33.177692 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.177666    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.181593 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.181565    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c0ec41\"/\"ipp-route-c0ec41-kserve-self-signed-certs\""
Sep 10 16:08:33.181768 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.181644    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c0ec41\"/\"default-dockercfg-kt7zb\""
Sep 10 16:08:33.181768 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.181698    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c0ec41\"/\"openshift-service-ca.crt\""
Sep 10 16:08:33.181768 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.181718    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c0ec41\"/\"kube-root-ca.crt\""
Sep 10 16:08:33.188699 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.188670    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"]
Sep 10 16:08:33.259608 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.259567    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-dshm\") pod \"ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") " pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.259608 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.259612    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz9jc\" (UniqueName: \"kubernetes.io/projected/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-kube-api-access-rz9jc\") pod \"ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") " pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.259868 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.259637    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-model-cache\") pod \"ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") " pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.259868 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.259759    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-tls-certs\") pod \"ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") " pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.259868 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.259792    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-tmp-dir\") pod \"ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") " pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.259975 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.259870    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-home\") pod \"ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") " pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.361060 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.361011    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-home\") pod \"ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") " pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.361290 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.361078    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-dshm\") pod \"ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") " pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.361290 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.361099    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz9jc\" (UniqueName: \"kubernetes.io/projected/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-kube-api-access-rz9jc\") pod \"ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") " pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.361290 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.361121    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-model-cache\") pod \"ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") " pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.361290 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.361170    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-tls-certs\") pod \"ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") " pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.361290 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.361192    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-tmp-dir\") pod \"ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") " pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.361627 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.361556    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-home\") pod \"ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") " pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.361627 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.361607    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-model-cache\") pod \"ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") " pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.361747 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.361643    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-tmp-dir\") pod \"ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") " pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.363570 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.363548    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-dshm\") pod \"ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") " pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.363776 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.363759    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-tls-certs\") pod \"ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") " pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.370237 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.370210    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz9jc\" (UniqueName: \"kubernetes.io/projected/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-kube-api-access-rz9jc\") pod \"ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") " pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.488720 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.488622    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:33.767528 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.767497    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"]
Sep 10 16:08:33.769059 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:08:33.769024    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd9d0c11_1c3f_46ee_989b_7ec21c9cad24.slice/crio-07b0bc38b99f374ca988084ecb1f4adfa8ba9761674e1c860b7fb01f58f79e1e WatchSource:0}: Error finding container 07b0bc38b99f374ca988084ecb1f4adfa8ba9761674e1c860b7fb01f58f79e1e: Status 404 returned error can't find the container with id 07b0bc38b99f374ca988084ecb1f4adfa8ba9761674e1c860b7fb01f58f79e1e
Sep 10 16:08:33.927557 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.927516    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh" event={"ID":"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24","Type":"ContainerStarted","Data":"bef302be366a9232b48c20b00e1b660d60072405217ace057815946f16dd51e4"}
Sep 10 16:08:33.927754 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.927567    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh" event={"ID":"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24","Type":"ContainerStarted","Data":"07b0bc38b99f374ca988084ecb1f4adfa8ba9761674e1c860b7fb01f58f79e1e"}
Sep 10 16:08:33.951254 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:33.951187    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh" podStartSLOduration=0.951169776 podStartE2EDuration="951.169776ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:08:33.949564811 +0000 UTC m=+1586.621974911" watchObservedRunningTime="2026-09-10 16:08:33.951169776 +0000 UTC m=+1586.623579912"
Sep 10 16:08:43.489374 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:43.489329    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:43.489792 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:43.489393    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:43.501869 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:43.501835    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:08:43.997056 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:08:43.997025    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:09:01.919817 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:01.919733    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"]
Sep 10 16:09:01.920453 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:01.920248    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" containerName="istio-proxy"
Sep 10 16:09:01.920453 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:01.920269    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2997d9f3-34a3-47b5-86dd-e4bb79bb0cd9" containerName="istio-proxy"
Sep 10 16:09:01.922760 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:01.922740    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"
Sep 10 16:09:01.925822 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:01.925794    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-044fdb-serving-cert\""
Sep 10 16:09:01.925972 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:01.925799    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 16:09:01.927076 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:01.927056    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 16:09:01.927297 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:01.927281    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nwclg\""
Sep 10 16:09:01.927391 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:01.927326    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:09:01.939229 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:01.939204    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"]
Sep 10 16:09:02.040652 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:02.040616    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc\" (UID: \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"
Sep 10 16:09:02.040844 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:02.040672    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5hbg\" (UniqueName: \"kubernetes.io/projected/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-kube-api-access-j5hbg\") pod \"maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc\" (UID: \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"
Sep 10 16:09:02.040844 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:02.040710    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc\" (UID: \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"
Sep 10 16:09:02.040844 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:02.040755    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc\" (UID: \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"
Sep 10 16:09:02.141461 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:02.141418    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc\" (UID: \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"
Sep 10 16:09:02.141671 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:02.141488    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5hbg\" (UniqueName: \"kubernetes.io/projected/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-kube-api-access-j5hbg\") pod \"maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc\" (UID: \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"
Sep 10 16:09:02.141671 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:02.141529    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc\" (UID: \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"
Sep 10 16:09:02.141671 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:02.141575    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc\" (UID: \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"
Sep 10 16:09:02.142193 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:02.142168    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc\" (UID: \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"
Sep 10 16:09:02.143963 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:02.143938    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc\" (UID: \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"
Sep 10 16:09:02.144058 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:02.143981    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc\" (UID: \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"
Sep 10 16:09:02.150889 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:02.150862    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5hbg\" (UniqueName: \"kubernetes.io/projected/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-kube-api-access-j5hbg\") pod \"maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc\" (UID: \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"
Sep 10 16:09:02.235665 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:02.235562    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"
Sep 10 16:09:02.518704 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:02.518679    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"]
Sep 10 16:09:02.521203 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:09:02.521166    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3b2454_4dfb_4752_bee3_8eba05ee78ea.slice/crio-32fe34a0db85b38af5fda753c1c8d67e988606f9e715e360d574339933a402fd WatchSource:0}: Error finding container 32fe34a0db85b38af5fda753c1c8d67e988606f9e715e360d574339933a402fd: Status 404 returned error can't find the container with id 32fe34a0db85b38af5fda753c1c8d67e988606f9e715e360d574339933a402fd
Sep 10 16:09:02.522628 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:02.522605    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 16:09:03.097451 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:03.097411    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc" event={"ID":"bf3b2454-4dfb-4752-bee3-8eba05ee78ea","Type":"ContainerStarted","Data":"32fe34a0db85b38af5fda753c1c8d67e988606f9e715e360d574339933a402fd"}
Sep 10 16:09:05.112525 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:05.112479    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc" event={"ID":"bf3b2454-4dfb-4752-bee3-8eba05ee78ea","Type":"ContainerStarted","Data":"c709108a51a7f2ca7866a21d74687b361f7cc3c037612173d2a3a1fdfef75129"}
Sep 10 16:09:05.112956 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:05.112554    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"
Sep 10 16:09:05.141238 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:05.141180    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc" podStartSLOduration=1.678385909 podStartE2EDuration="4.141161281s" totalImagesPullingTime="2.462775372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:09:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:09:05.139995583 +0000 UTC m=+1617.812405665" watchObservedRunningTime="2026-09-10 16:09:05.141161281 +0000 UTC m=+1617.813571364"
Sep 10 16:09:11.124716 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:11.124681    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"
Sep 10 16:09:32.594915 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:32.594878    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"]
Sep 10 16:09:32.595431 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:32.595091    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc" podUID="bf3b2454-4dfb-4752-bee3-8eba05ee78ea" containerName="maas-api" containerID="cri-o://c709108a51a7f2ca7866a21d74687b361f7cc3c037612173d2a3a1fdfef75129" gracePeriod=30
Sep 10 16:09:32.850159 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:32.850081    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"
Sep 10 16:09:32.944609 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:32.944572    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5hbg\" (UniqueName: \"kubernetes.io/projected/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-kube-api-access-j5hbg\") pod \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\" (UID: \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\") "
Sep 10 16:09:32.944809 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:32.944674    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-openshift-service-ca\") pod \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\" (UID: \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\") "
Sep 10 16:09:32.944809 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:32.944718    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-maas-api-tls\") pod \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\" (UID: \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\") "
Sep 10 16:09:32.944809 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:32.944735    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-metrics-certs\") pod \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\" (UID: \"bf3b2454-4dfb-4752-bee3-8eba05ee78ea\") "
Sep 10 16:09:32.945130 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:32.945094    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bf3b2454-4dfb-4752-bee3-8eba05ee78ea" (UID: "bf3b2454-4dfb-4752-bee3-8eba05ee78ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:09:32.947281 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:32.947251    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-kube-api-access-j5hbg" (OuterVolumeSpecName: "kube-api-access-j5hbg") pod "bf3b2454-4dfb-4752-bee3-8eba05ee78ea" (UID: "bf3b2454-4dfb-4752-bee3-8eba05ee78ea"). InnerVolumeSpecName "kube-api-access-j5hbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:09:32.947433 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:32.947340    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bf3b2454-4dfb-4752-bee3-8eba05ee78ea" (UID: "bf3b2454-4dfb-4752-bee3-8eba05ee78ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:09:32.947495 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:32.947433    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bf3b2454-4dfb-4752-bee3-8eba05ee78ea" (UID: "bf3b2454-4dfb-4752-bee3-8eba05ee78ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:09:33.046324 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:33.046268    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5hbg\" (UniqueName: \"kubernetes.io/projected/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-kube-api-access-j5hbg\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:09:33.046324 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:33.046329    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-openshift-service-ca\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:09:33.046639 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:33.046340    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-maas-api-tls\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:09:33.046639 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:33.046353    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf3b2454-4dfb-4752-bee3-8eba05ee78ea-metrics-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:09:33.275835 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:33.275735    2572 generic.go:356] "Generic (PLEG): container finished" podID="bf3b2454-4dfb-4752-bee3-8eba05ee78ea" containerID="c709108a51a7f2ca7866a21d74687b361f7cc3c037612173d2a3a1fdfef75129" exitCode=0
Sep 10 16:09:33.275835 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:33.275784    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"
Sep 10 16:09:33.275835 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:33.275818    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc" event={"ID":"bf3b2454-4dfb-4752-bee3-8eba05ee78ea","Type":"ContainerDied","Data":"c709108a51a7f2ca7866a21d74687b361f7cc3c037612173d2a3a1fdfef75129"}
Sep 10 16:09:33.276105 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:33.275878    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc" event={"ID":"bf3b2454-4dfb-4752-bee3-8eba05ee78ea","Type":"ContainerDied","Data":"32fe34a0db85b38af5fda753c1c8d67e988606f9e715e360d574339933a402fd"}
Sep 10 16:09:33.276105 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:33.275909    2572 scope.go:117] "RemoveContainer" containerID="c709108a51a7f2ca7866a21d74687b361f7cc3c037612173d2a3a1fdfef75129"
Sep 10 16:09:33.286257 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:33.286234    2572 scope.go:117] "RemoveContainer" containerID="c709108a51a7f2ca7866a21d74687b361f7cc3c037612173d2a3a1fdfef75129"
Sep 10 16:09:33.286618 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:09:33.286595    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c709108a51a7f2ca7866a21d74687b361f7cc3c037612173d2a3a1fdfef75129\": container with ID starting with c709108a51a7f2ca7866a21d74687b361f7cc3c037612173d2a3a1fdfef75129 not found: ID does not exist" containerID="c709108a51a7f2ca7866a21d74687b361f7cc3c037612173d2a3a1fdfef75129"
Sep 10 16:09:33.286678 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:33.286636    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c709108a51a7f2ca7866a21d74687b361f7cc3c037612173d2a3a1fdfef75129"} err="failed to get container status \"c709108a51a7f2ca7866a21d74687b361f7cc3c037612173d2a3a1fdfef75129\": rpc error: code = NotFound desc = could not find container \"c709108a51a7f2ca7866a21d74687b361f7cc3c037612173d2a3a1fdfef75129\": container with ID starting with c709108a51a7f2ca7866a21d74687b361f7cc3c037612173d2a3a1fdfef75129 not found: ID does not exist"
Sep 10 16:09:33.300483 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:33.300450    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"]
Sep 10 16:09:33.304341 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:33.304311    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-044fdb-974f4ffcd-8nsvc"]
Sep 10 16:09:33.994545 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:33.994505    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf3b2454-4dfb-4752-bee3-8eba05ee78ea" path="/var/lib/kubelet/pods/bf3b2454-4dfb-4752-bee3-8eba05ee78ea/volumes"
Sep 10 16:09:43.521928 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:43.521882    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0ec41-5rfnj"]
Sep 10 16:09:43.522503 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:43.522484    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf3b2454-4dfb-4752-bee3-8eba05ee78ea" containerName="maas-api"
Sep 10 16:09:43.522558 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:43.522507    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf3b2454-4dfb-4752-bee3-8eba05ee78ea" containerName="maas-api"
Sep 10 16:09:43.522614 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:43.522603    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf3b2454-4dfb-4752-bee3-8eba05ee78ea" containerName="maas-api"
Sep 10 16:09:43.525011 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:43.524989    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0ec41-5rfnj"
Sep 10 16:09:43.527423 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:43.527398    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\""
Sep 10 16:09:43.527576 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:43.527402    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:09:43.532621 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:43.532592    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0ec41-5rfnj"]
Sep 10 16:09:43.648502 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:43.648455    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/553f1a53-949d-4634-8e5f-56b795073384-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c0ec41-5rfnj\" (UID: \"553f1a53-949d-4634-8e5f-56b795073384\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0ec41-5rfnj"
Sep 10 16:09:43.749919 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:43.749875    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/553f1a53-949d-4634-8e5f-56b795073384-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c0ec41-5rfnj\" (UID: \"553f1a53-949d-4634-8e5f-56b795073384\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0ec41-5rfnj"
Sep 10 16:09:43.750607 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:43.750582    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/553f1a53-949d-4634-8e5f-56b795073384-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c0ec41-5rfnj\" (UID: \"553f1a53-949d-4634-8e5f-56b795073384\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0ec41-5rfnj"
Sep 10 16:09:43.837088 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:43.837046    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0ec41-5rfnj"
Sep 10 16:09:44.112408 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:44.112381    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0ec41-5rfnj"]
Sep 10 16:09:44.113371 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:09:44.113345    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553f1a53_949d_4634_8e5f_56b795073384.slice/crio-a26a087c84c61d38600ad9956e20f8a867d54e8e91346165d9dc8270442ccabc WatchSource:0}: Error finding container a26a087c84c61d38600ad9956e20f8a867d54e8e91346165d9dc8270442ccabc: Status 404 returned error can't find the container with id a26a087c84c61d38600ad9956e20f8a867d54e8e91346165d9dc8270442ccabc
Sep 10 16:09:44.347078 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:44.347037    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0ec41-5rfnj" event={"ID":"553f1a53-949d-4634-8e5f-56b795073384","Type":"ContainerStarted","Data":"a26a087c84c61d38600ad9956e20f8a867d54e8e91346165d9dc8270442ccabc"}
Sep 10 16:09:45.354971 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:45.354932    2572 generic.go:356] "Generic (PLEG): container finished" podID="553f1a53-949d-4634-8e5f-56b795073384" containerID="fffbe0ac7e75ed2574520905d1508d5d05b8ab068aac5990c6ec82aced36e1f7" exitCode=0
Sep 10 16:09:45.355390 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:45.354997    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0ec41-5rfnj" event={"ID":"553f1a53-949d-4634-8e5f-56b795073384","Type":"ContainerDied","Data":"fffbe0ac7e75ed2574520905d1508d5d05b8ab068aac5990c6ec82aced36e1f7"}
Sep 10 16:09:46.500978 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:46.500940    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0ec41-5rfnj"
Sep 10 16:09:46.578766 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:46.578722    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/553f1a53-949d-4634-8e5f-56b795073384-maas-api-service-ca\") pod \"553f1a53-949d-4634-8e5f-56b795073384\" (UID: \"553f1a53-949d-4634-8e5f-56b795073384\") "
Sep 10 16:09:46.579112 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:46.579087    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/553f1a53-949d-4634-8e5f-56b795073384-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "553f1a53-949d-4634-8e5f-56b795073384" (UID: "553f1a53-949d-4634-8e5f-56b795073384"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:09:46.680044 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:46.679956    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/553f1a53-949d-4634-8e5f-56b795073384-maas-api-service-ca\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:09:47.370082 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:47.370051    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0ec41-5rfnj"
Sep 10 16:09:47.370270 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:47.370088    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0ec41-5rfnj" event={"ID":"553f1a53-949d-4634-8e5f-56b795073384","Type":"ContainerDied","Data":"a26a087c84c61d38600ad9956e20f8a867d54e8e91346165d9dc8270442ccabc"}
Sep 10 16:09:47.370270 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:47.370138    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a26a087c84c61d38600ad9956e20f8a867d54e8e91346165d9dc8270442ccabc"
Sep 10 16:09:59.612815 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:59.612757    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0ec41-5rfnj"]
Sep 10 16:09:59.616707 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:59.616679    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c0ec41-5rfnj"]
Sep 10 16:09:59.993023 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:09:59.992921    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="553f1a53-949d-4634-8e5f-56b795073384" path="/var/lib/kubelet/pods/553f1a53-949d-4634-8e5f-56b795073384/volumes"
Sep 10 16:10:05.001482 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:05.001442    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"]
Sep 10 16:10:20.021693 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.021630    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh" podUID="fd9d0c11-1c3f-46ee-989b-7ec21c9cad24" containerName="main" containerID="cri-o://bef302be366a9232b48c20b00e1b660d60072405217ace057815946f16dd51e4" gracePeriod=45
Sep 10 16:10:20.272955 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.272891    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:10:20.403135 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.403096    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-model-cache\") pod \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") "
Sep 10 16:10:20.403363 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.403155    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-dshm\") pod \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") "
Sep 10 16:10:20.403363 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.403184    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-tls-certs\") pod \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") "
Sep 10 16:10:20.403363 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.403223    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-tmp-dir\") pod \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") "
Sep 10 16:10:20.403363 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.403280    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz9jc\" (UniqueName: \"kubernetes.io/projected/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-kube-api-access-rz9jc\") pod \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") "
Sep 10 16:10:20.403363 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.403342    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-home\") pod \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\" (UID: \"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24\") "
Sep 10 16:10:20.403678 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.403413    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd9d0c11-1c3f-46ee-989b-7ec21c9cad24" (UID: "fd9d0c11-1c3f-46ee-989b-7ec21c9cad24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:10:20.403746 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.403679    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd9d0c11-1c3f-46ee-989b-7ec21c9cad24" (UID: "fd9d0c11-1c3f-46ee-989b-7ec21c9cad24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:10:20.403746 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.403706    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-model-cache\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:10:20.403926 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.403848    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-home" (OuterVolumeSpecName: "home") pod "fd9d0c11-1c3f-46ee-989b-7ec21c9cad24" (UID: "fd9d0c11-1c3f-46ee-989b-7ec21c9cad24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:10:20.405779 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.405754    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd9d0c11-1c3f-46ee-989b-7ec21c9cad24" (UID: "fd9d0c11-1c3f-46ee-989b-7ec21c9cad24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:10:20.405862 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.405783    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-dshm" (OuterVolumeSpecName: "dshm") pod "fd9d0c11-1c3f-46ee-989b-7ec21c9cad24" (UID: "fd9d0c11-1c3f-46ee-989b-7ec21c9cad24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:10:20.405862 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.405827    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-kube-api-access-rz9jc" (OuterVolumeSpecName: "kube-api-access-rz9jc") pod "fd9d0c11-1c3f-46ee-989b-7ec21c9cad24" (UID: "fd9d0c11-1c3f-46ee-989b-7ec21c9cad24"). InnerVolumeSpecName "kube-api-access-rz9jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:10:20.504288 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.504247    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz9jc\" (UniqueName: \"kubernetes.io/projected/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-kube-api-access-rz9jc\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:10:20.504288 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.504282    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-home\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:10:20.504288 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.504294    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-dshm\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:10:20.504569 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.504331    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-tls-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:10:20.504569 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.504344    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24-tmp-dir\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:10:20.572068 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.572026    2572 generic.go:356] "Generic (PLEG): container finished" podID="fd9d0c11-1c3f-46ee-989b-7ec21c9cad24" containerID="bef302be366a9232b48c20b00e1b660d60072405217ace057815946f16dd51e4" exitCode=0
Sep 10 16:10:20.572068 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.572069    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"
Sep 10 16:10:20.572283 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.572114    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh" event={"ID":"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24","Type":"ContainerDied","Data":"bef302be366a9232b48c20b00e1b660d60072405217ace057815946f16dd51e4"}
Sep 10 16:10:20.572283 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.572158    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh" event={"ID":"fd9d0c11-1c3f-46ee-989b-7ec21c9cad24","Type":"ContainerDied","Data":"07b0bc38b99f374ca988084ecb1f4adfa8ba9761674e1c860b7fb01f58f79e1e"}
Sep 10 16:10:20.572283 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.572177    2572 scope.go:117] "RemoveContainer" containerID="bef302be366a9232b48c20b00e1b660d60072405217ace057815946f16dd51e4"
Sep 10 16:10:20.582032 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.582014    2572 scope.go:117] "RemoveContainer" containerID="bef302be366a9232b48c20b00e1b660d60072405217ace057815946f16dd51e4"
Sep 10 16:10:20.582372 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:10:20.582342    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef302be366a9232b48c20b00e1b660d60072405217ace057815946f16dd51e4\": container with ID starting with bef302be366a9232b48c20b00e1b660d60072405217ace057815946f16dd51e4 not found: ID does not exist" containerID="bef302be366a9232b48c20b00e1b660d60072405217ace057815946f16dd51e4"
Sep 10 16:10:20.582459 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.582381    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef302be366a9232b48c20b00e1b660d60072405217ace057815946f16dd51e4"} err="failed to get container status \"bef302be366a9232b48c20b00e1b660d60072405217ace057815946f16dd51e4\": rpc error: code = NotFound desc = could not find container \"bef302be366a9232b48c20b00e1b660d60072405217ace057815946f16dd51e4\": container with ID starting with bef302be366a9232b48c20b00e1b660d60072405217ace057815946f16dd51e4 not found: ID does not exist"
Sep 10 16:10:20.595525 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.595496    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"]
Sep 10 16:10:20.602417 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:20.602390    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-c0ec41/ipp-route-c0ec41-kserve-ff95bbbc5-g4pqh"]
Sep 10 16:10:21.992051 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:21.992005    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd9d0c11-1c3f-46ee-989b-7ec21c9cad24" path="/var/lib/kubelet/pods/fd9d0c11-1c3f-46ee-989b-7ec21c9cad24/volumes"
Sep 10 16:10:49.858778 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:49.858681    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"]
Sep 10 16:10:49.859396 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:49.858953    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6" podUID="b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" containerName="istio-proxy" containerID="cri-o://928aa9ce66c336810d8b0ad18a8aa3b585e12233e67eb306872dc5a9c47e5a8c" gracePeriod=30
Sep 10 16:10:51.672733 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:51.672693    2572 patch_prober.go:28] interesting pod/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 16:10:51.673156 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:51.672766    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6" podUID="b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 16:10:55.116148 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.116114    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:10:55.238097 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.237989    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-workload-socket\") pod \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") "
Sep 10 16:10:55.238097 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.238048    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-credential-socket\") pod \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") "
Sep 10 16:10:55.238097 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.238092    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56986\" (UniqueName: \"kubernetes.io/projected/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-kube-api-access-56986\") pod \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") "
Sep 10 16:10:55.238459 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.238125    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istiod-ca-cert\") pod \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") "
Sep 10 16:10:55.238459 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.238168    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-envoy\") pod \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") "
Sep 10 16:10:55.238459 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.238208    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-workload-certs\") pod \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") "
Sep 10 16:10:55.238459 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.238256    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-token\") pod \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") "
Sep 10 16:10:55.238459 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.238283    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-podinfo\") pod \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") "
Sep 10 16:10:55.238459 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.238346    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-data\") pod \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\" (UID: \"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e\") "
Sep 10 16:10:55.238459 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.238375    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" (UID: "b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:10:55.238818 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.238482    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" (UID: "b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:10:55.238818 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.238542    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" (UID: "b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:10:55.238818 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.238684    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-data" (OuterVolumeSpecName: "istio-data") pod "b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" (UID: "b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:10:55.238818 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.238714    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-workload-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:10:55.238818 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.238614    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" (UID: "b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:10:55.238818 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.238727    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-credential-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:10:55.238818 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.238760    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-workload-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:10:55.240546 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.240505    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" (UID: "b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:10:55.240685 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.240595    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-token" (OuterVolumeSpecName: "istio-token") pod "b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" (UID: "b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:10:55.240685 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.240604    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" (UID: "b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:10:55.240685 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.240622    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-kube-api-access-56986" (OuterVolumeSpecName: "kube-api-access-56986") pod "b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" (UID: "b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e"). InnerVolumeSpecName "kube-api-access-56986". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:10:55.340091 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.340047    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-token\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:10:55.340091 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.340080    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-podinfo\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:10:55.340091 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.340089    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-data\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:10:55.340091 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.340098    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56986\" (UniqueName: \"kubernetes.io/projected/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-kube-api-access-56986\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:10:55.340091 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.340107    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istiod-ca-cert\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:10:55.340446 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.340116    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e-istio-envoy\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:10:55.781287 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.781251    2572 generic.go:356] "Generic (PLEG): container finished" podID="b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" containerID="928aa9ce66c336810d8b0ad18a8aa3b585e12233e67eb306872dc5a9c47e5a8c" exitCode=0
Sep 10 16:10:55.781518 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.781331    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"
Sep 10 16:10:55.781518 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.781350    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6" event={"ID":"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e","Type":"ContainerDied","Data":"928aa9ce66c336810d8b0ad18a8aa3b585e12233e67eb306872dc5a9c47e5a8c"}
Sep 10 16:10:55.781518 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.781418    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6" event={"ID":"b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e","Type":"ContainerDied","Data":"3941c68c9c109ccac19ad9f8f4f71ba107017694a28c09a7ecadd9364d09a0b7"}
Sep 10 16:10:55.781518 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.781449    2572 scope.go:117] "RemoveContainer" containerID="928aa9ce66c336810d8b0ad18a8aa3b585e12233e67eb306872dc5a9c47e5a8c"
Sep 10 16:10:55.792025 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.792005    2572 scope.go:117] "RemoveContainer" containerID="edf0499c5c3bda1b57ba0a03356d10b3cb5869450a0b4ce24628d82916e1097b"
Sep 10 16:10:55.800873 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.800853    2572 scope.go:117] "RemoveContainer" containerID="928aa9ce66c336810d8b0ad18a8aa3b585e12233e67eb306872dc5a9c47e5a8c"
Sep 10 16:10:55.801226 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:10:55.801185    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"928aa9ce66c336810d8b0ad18a8aa3b585e12233e67eb306872dc5a9c47e5a8c\": container with ID starting with 928aa9ce66c336810d8b0ad18a8aa3b585e12233e67eb306872dc5a9c47e5a8c not found: ID does not exist" containerID="928aa9ce66c336810d8b0ad18a8aa3b585e12233e67eb306872dc5a9c47e5a8c"
Sep 10 16:10:55.801420 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.801229    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"928aa9ce66c336810d8b0ad18a8aa3b585e12233e67eb306872dc5a9c47e5a8c"} err="failed to get container status \"928aa9ce66c336810d8b0ad18a8aa3b585e12233e67eb306872dc5a9c47e5a8c\": rpc error: code = NotFound desc = could not find container \"928aa9ce66c336810d8b0ad18a8aa3b585e12233e67eb306872dc5a9c47e5a8c\": container with ID starting with 928aa9ce66c336810d8b0ad18a8aa3b585e12233e67eb306872dc5a9c47e5a8c not found: ID does not exist"
Sep 10 16:10:55.801420 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.801254    2572 scope.go:117] "RemoveContainer" containerID="edf0499c5c3bda1b57ba0a03356d10b3cb5869450a0b4ce24628d82916e1097b"
Sep 10 16:10:55.801901 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:10:55.801872    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edf0499c5c3bda1b57ba0a03356d10b3cb5869450a0b4ce24628d82916e1097b\": container with ID starting with edf0499c5c3bda1b57ba0a03356d10b3cb5869450a0b4ce24628d82916e1097b not found: ID does not exist" containerID="edf0499c5c3bda1b57ba0a03356d10b3cb5869450a0b4ce24628d82916e1097b"
Sep 10 16:10:55.802010 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.801910    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edf0499c5c3bda1b57ba0a03356d10b3cb5869450a0b4ce24628d82916e1097b"} err="failed to get container status \"edf0499c5c3bda1b57ba0a03356d10b3cb5869450a0b4ce24628d82916e1097b\": rpc error: code = NotFound desc = could not find container \"edf0499c5c3bda1b57ba0a03356d10b3cb5869450a0b4ce24628d82916e1097b\": container with ID starting with edf0499c5c3bda1b57ba0a03356d10b3cb5869450a0b4ce24628d82916e1097b not found: ID does not exist"
Sep 10 16:10:55.804466 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.804446    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"]
Sep 10 16:10:55.807889 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.807865    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c0ec41-openshift-default-57d68d6d5d-d92z6"]
Sep 10 16:10:55.992595 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:10:55.992558    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" path="/var/lib/kubelet/pods/b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e/volumes"
Sep 10 16:11:08.378657 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.378618    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-50ed73-bsdgq"]
Sep 10 16:11:08.379062 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.378971    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" containerName="istio-proxy"
Sep 10 16:11:08.379062 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.378983    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" containerName="istio-proxy"
Sep 10 16:11:08.379062 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.378994    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="553f1a53-949d-4634-8e5f-56b795073384" containerName="revoke-keys"
Sep 10 16:11:08.379062 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.379002    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="553f1a53-949d-4634-8e5f-56b795073384" containerName="revoke-keys"
Sep 10 16:11:08.379062 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.379017    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd9d0c11-1c3f-46ee-989b-7ec21c9cad24" containerName="main"
Sep 10 16:11:08.379062 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.379023    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd9d0c11-1c3f-46ee-989b-7ec21c9cad24" containerName="main"
Sep 10 16:11:08.379062 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.379033    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" containerName="istio-proxy"
Sep 10 16:11:08.379062 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.379038    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" containerName="istio-proxy"
Sep 10 16:11:08.379416 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.379089    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="553f1a53-949d-4634-8e5f-56b795073384" containerName="revoke-keys"
Sep 10 16:11:08.379416 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.379096    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" containerName="istio-proxy"
Sep 10 16:11:08.379416 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.379104    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b966694f-6e28-4ac8-b6f3-bd18b9f1cd8e" containerName="istio-proxy"
Sep 10 16:11:08.379416 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.379111    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd9d0c11-1c3f-46ee-989b-7ec21c9cad24" containerName="main"
Sep 10 16:11:08.382136 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.382116    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-50ed73-bsdgq"
Sep 10 16:11:08.384791 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.384768    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kwwht\""
Sep 10 16:11:08.385174 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.385159    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:11:08.393054 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.393022    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-50ed73-bsdgq"]
Sep 10 16:11:08.459885 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.459840    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8606646b-7554-44de-9672-e9c8182782fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-50ed73-bsdgq\" (UID: \"8606646b-7554-44de-9672-e9c8182782fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-50ed73-bsdgq"
Sep 10 16:11:08.561066 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.561018    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8606646b-7554-44de-9672-e9c8182782fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-50ed73-bsdgq\" (UID: \"8606646b-7554-44de-9672-e9c8182782fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-50ed73-bsdgq"
Sep 10 16:11:08.561755 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.561735    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8606646b-7554-44de-9672-e9c8182782fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-50ed73-bsdgq\" (UID: \"8606646b-7554-44de-9672-e9c8182782fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-50ed73-bsdgq"
Sep 10 16:11:08.694445 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.694342    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-50ed73-bsdgq"
Sep 10 16:11:08.982804 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:08.982447    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-50ed73-bsdgq"]
Sep 10 16:11:08.986150 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:11:08.986104    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8606646b_7554_44de_9672_e9c8182782fa.slice/crio-e7a7e72c13b6d6f6be70110ac0d64561c71fb7a387c4f346bfa2b7da91209d62 WatchSource:0}: Error finding container e7a7e72c13b6d6f6be70110ac0d64561c71fb7a387c4f346bfa2b7da91209d62: Status 404 returned error can't find the container with id e7a7e72c13b6d6f6be70110ac0d64561c71fb7a387c4f346bfa2b7da91209d62
Sep 10 16:11:09.863078 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:09.863039    2572 generic.go:356] "Generic (PLEG): container finished" podID="8606646b-7554-44de-9672-e9c8182782fa" containerID="b897ec3ba02027c90b48c6abafa459dab00f18c8a16bbc861f04fdd0a3b4fd26" exitCode=0
Sep 10 16:11:09.863524 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:09.863130    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-50ed73-bsdgq" event={"ID":"8606646b-7554-44de-9672-e9c8182782fa","Type":"ContainerDied","Data":"b897ec3ba02027c90b48c6abafa459dab00f18c8a16bbc861f04fdd0a3b4fd26"}
Sep 10 16:11:09.863524 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:09.863185    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-50ed73-bsdgq" event={"ID":"8606646b-7554-44de-9672-e9c8182782fa","Type":"ContainerStarted","Data":"e7a7e72c13b6d6f6be70110ac0d64561c71fb7a387c4f346bfa2b7da91209d62"}
Sep 10 16:11:10.999858 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:10.999832    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-50ed73-bsdgq"
Sep 10 16:11:11.083558 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:11.083520    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8606646b-7554-44de-9672-e9c8182782fa-maas-api-service-ca\") pod \"8606646b-7554-44de-9672-e9c8182782fa\" (UID: \"8606646b-7554-44de-9672-e9c8182782fa\") "
Sep 10 16:11:11.083910 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:11.083880    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8606646b-7554-44de-9672-e9c8182782fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8606646b-7554-44de-9672-e9c8182782fa" (UID: "8606646b-7554-44de-9672-e9c8182782fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:11:11.184660 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:11.184553    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8606646b-7554-44de-9672-e9c8182782fa-maas-api-service-ca\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:11:11.883097 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:11.883067    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-50ed73-bsdgq"
Sep 10 16:11:11.883290 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:11.883100    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-50ed73-bsdgq" event={"ID":"8606646b-7554-44de-9672-e9c8182782fa","Type":"ContainerDied","Data":"e7a7e72c13b6d6f6be70110ac0d64561c71fb7a387c4f346bfa2b7da91209d62"}
Sep 10 16:11:11.883290 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:11.883169    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7a7e72c13b6d6f6be70110ac0d64561c71fb7a387c4f346bfa2b7da91209d62"
Sep 10 16:11:19.224580 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:19.224499    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-50ed73-bsdgq"]
Sep 10 16:11:19.228205 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:19.228167    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-50ed73-bsdgq"]
Sep 10 16:11:19.992250 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:19.992212    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8606646b-7554-44de-9672-e9c8182782fa" path="/var/lib/kubelet/pods/8606646b-7554-44de-9672-e9c8182782fa/volumes"
Sep 10 16:11:26.261116 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:26.261073    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"]
Sep 10 16:11:26.261670 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:26.261325    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms" podUID="51c87e36-1cc9-4711-8220-e379dbd36f30" containerName="istio-proxy" containerID="cri-o://2fb117b983e26a1ef412a7ce7355329e127732f972ad84dadad6f53ebddbb404" gracePeriod=30
Sep 10 16:11:31.512487 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.512456    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 16:11:31.576947 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.576907    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x994c\" (UniqueName: \"kubernetes.io/projected/51c87e36-1cc9-4711-8220-e379dbd36f30-kube-api-access-x994c\") pod \"51c87e36-1cc9-4711-8220-e379dbd36f30\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") "
Sep 10 16:11:31.576947 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.576951    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c87e36-1cc9-4711-8220-e379dbd36f30-istiod-ca-cert\") pod \"51c87e36-1cc9-4711-8220-e379dbd36f30\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") "
Sep 10 16:11:31.577207 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.577005    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-token\") pod \"51c87e36-1cc9-4711-8220-e379dbd36f30\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") "
Sep 10 16:11:31.577207 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.577023    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-podinfo\") pod \"51c87e36-1cc9-4711-8220-e379dbd36f30\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") "
Sep 10 16:11:31.577207 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.577065    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-workload-certs\") pod \"51c87e36-1cc9-4711-8220-e379dbd36f30\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") "
Sep 10 16:11:31.577207 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.577117    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-workload-socket\") pod \"51c87e36-1cc9-4711-8220-e379dbd36f30\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") "
Sep 10 16:11:31.577207 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.577143    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-data\") pod \"51c87e36-1cc9-4711-8220-e379dbd36f30\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") "
Sep 10 16:11:31.577207 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.577171    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-envoy\") pod \"51c87e36-1cc9-4711-8220-e379dbd36f30\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") "
Sep 10 16:11:31.577207 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.577197    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-credential-socket\") pod \"51c87e36-1cc9-4711-8220-e379dbd36f30\" (UID: \"51c87e36-1cc9-4711-8220-e379dbd36f30\") "
Sep 10 16:11:31.577630 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.577403    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "51c87e36-1cc9-4711-8220-e379dbd36f30" (UID: "51c87e36-1cc9-4711-8220-e379dbd36f30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:11:31.577630 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.577422    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "51c87e36-1cc9-4711-8220-e379dbd36f30" (UID: "51c87e36-1cc9-4711-8220-e379dbd36f30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:11:31.577630 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.577489    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51c87e36-1cc9-4711-8220-e379dbd36f30-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "51c87e36-1cc9-4711-8220-e379dbd36f30" (UID: "51c87e36-1cc9-4711-8220-e379dbd36f30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:11:31.577630 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.577593    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c87e36-1cc9-4711-8220-e379dbd36f30-istiod-ca-cert\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:11:31.577630 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.577606    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "51c87e36-1cc9-4711-8220-e379dbd36f30" (UID: "51c87e36-1cc9-4711-8220-e379dbd36f30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:11:31.577630 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.577613    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-workload-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:11:31.577630 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.577630    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-data" (OuterVolumeSpecName: "istio-data") pod "51c87e36-1cc9-4711-8220-e379dbd36f30" (UID: "51c87e36-1cc9-4711-8220-e379dbd36f30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:11:31.578025 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.577645    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-workload-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:11:31.579359 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.579329    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c87e36-1cc9-4711-8220-e379dbd36f30-kube-api-access-x994c" (OuterVolumeSpecName: "kube-api-access-x994c") pod "51c87e36-1cc9-4711-8220-e379dbd36f30" (UID: "51c87e36-1cc9-4711-8220-e379dbd36f30"). InnerVolumeSpecName "kube-api-access-x994c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:11:31.579498 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.579442    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "51c87e36-1cc9-4711-8220-e379dbd36f30" (UID: "51c87e36-1cc9-4711-8220-e379dbd36f30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:11:31.579498 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.579448    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-token" (OuterVolumeSpecName: "istio-token") pod "51c87e36-1cc9-4711-8220-e379dbd36f30" (UID: "51c87e36-1cc9-4711-8220-e379dbd36f30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:11:31.579498 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.579446    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "51c87e36-1cc9-4711-8220-e379dbd36f30" (UID: "51c87e36-1cc9-4711-8220-e379dbd36f30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:11:31.678277 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.678233    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-data\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:11:31.678277 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.678268    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-envoy\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:11:31.678277 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.678281    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c87e36-1cc9-4711-8220-e379dbd36f30-credential-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:11:31.678571 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.678294    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x994c\" (UniqueName: \"kubernetes.io/projected/51c87e36-1cc9-4711-8220-e379dbd36f30-kube-api-access-x994c\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:11:31.678571 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.678331    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-token\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:11:31.678571 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:31.678342    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c87e36-1cc9-4711-8220-e379dbd36f30-istio-podinfo\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:11:32.000873 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:32.000835    2572 generic.go:356] "Generic (PLEG): container finished" podID="51c87e36-1cc9-4711-8220-e379dbd36f30" containerID="2fb117b983e26a1ef412a7ce7355329e127732f972ad84dadad6f53ebddbb404" exitCode=0
Sep 10 16:11:32.001002 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:32.000879    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"
Sep 10 16:11:32.001002 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:32.000952    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms" event={"ID":"51c87e36-1cc9-4711-8220-e379dbd36f30","Type":"ContainerDied","Data":"2fb117b983e26a1ef412a7ce7355329e127732f972ad84dadad6f53ebddbb404"}
Sep 10 16:11:32.001002 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:32.000999    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms" event={"ID":"51c87e36-1cc9-4711-8220-e379dbd36f30","Type":"ContainerDied","Data":"f13414d505301a5100bb549c96b38c3477cbe2ccee1786fbaaa53cb403c8cbf8"}
Sep 10 16:11:32.001149 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:32.001021    2572 scope.go:117] "RemoveContainer" containerID="2fb117b983e26a1ef412a7ce7355329e127732f972ad84dadad6f53ebddbb404"
Sep 10 16:11:32.010894 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:32.010871    2572 scope.go:117] "RemoveContainer" containerID="da8000335c6e84df461cfa56843eeb5af2edf84a4d4dd4502905cbf94506642b"
Sep 10 16:11:32.018774 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:32.018734    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"]
Sep 10 16:11:32.022289 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:32.022222    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-50ed73-openshift-default-6d67f5c548-kcpms"]
Sep 10 16:11:32.024210 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:32.024185    2572 scope.go:117] "RemoveContainer" containerID="2fb117b983e26a1ef412a7ce7355329e127732f972ad84dadad6f53ebddbb404"
Sep 10 16:11:32.024705 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:11:32.024684    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fb117b983e26a1ef412a7ce7355329e127732f972ad84dadad6f53ebddbb404\": container with ID starting with 2fb117b983e26a1ef412a7ce7355329e127732f972ad84dadad6f53ebddbb404 not found: ID does not exist" containerID="2fb117b983e26a1ef412a7ce7355329e127732f972ad84dadad6f53ebddbb404"
Sep 10 16:11:32.024782 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:32.024716    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb117b983e26a1ef412a7ce7355329e127732f972ad84dadad6f53ebddbb404"} err="failed to get container status \"2fb117b983e26a1ef412a7ce7355329e127732f972ad84dadad6f53ebddbb404\": rpc error: code = NotFound desc = could not find container \"2fb117b983e26a1ef412a7ce7355329e127732f972ad84dadad6f53ebddbb404\": container with ID starting with 2fb117b983e26a1ef412a7ce7355329e127732f972ad84dadad6f53ebddbb404 not found: ID does not exist"
Sep 10 16:11:32.024782 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:32.024736    2572 scope.go:117] "RemoveContainer" containerID="da8000335c6e84df461cfa56843eeb5af2edf84a4d4dd4502905cbf94506642b"
Sep 10 16:11:32.024998 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:11:32.024983    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da8000335c6e84df461cfa56843eeb5af2edf84a4d4dd4502905cbf94506642b\": container with ID starting with da8000335c6e84df461cfa56843eeb5af2edf84a4d4dd4502905cbf94506642b not found: ID does not exist" containerID="da8000335c6e84df461cfa56843eeb5af2edf84a4d4dd4502905cbf94506642b"
Sep 10 16:11:32.025044 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:32.025002    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da8000335c6e84df461cfa56843eeb5af2edf84a4d4dd4502905cbf94506642b"} err="failed to get container status \"da8000335c6e84df461cfa56843eeb5af2edf84a4d4dd4502905cbf94506642b\": rpc error: code = NotFound desc = could not find container \"da8000335c6e84df461cfa56843eeb5af2edf84a4d4dd4502905cbf94506642b\": container with ID starting with da8000335c6e84df461cfa56843eeb5af2edf84a4d4dd4502905cbf94506642b not found: ID does not exist"
Sep 10 16:11:33.992043 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:33.992007    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51c87e36-1cc9-4711-8220-e379dbd36f30" path="/var/lib/kubelet/pods/51c87e36-1cc9-4711-8220-e379dbd36f30/volumes"
Sep 10 16:11:44.919176 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:44.919127    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"]
Sep 10 16:11:44.919732 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:44.919416    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt" podUID="111222db-994b-41f5-a88d-56876c84d47e" containerName="istio-proxy" containerID="cri-o://09afb2dcd3dcfe607a3c04ada982bef50b4f80e961ead5fbcedae29adc211022" gracePeriod=30
Sep 10 16:11:50.109064 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.109031    2572 generic.go:356] "Generic (PLEG): container finished" podID="111222db-994b-41f5-a88d-56876c84d47e" containerID="09afb2dcd3dcfe607a3c04ada982bef50b4f80e961ead5fbcedae29adc211022" exitCode=0
Sep 10 16:11:50.109515 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.109107    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt" event={"ID":"111222db-994b-41f5-a88d-56876c84d47e","Type":"ContainerDied","Data":"09afb2dcd3dcfe607a3c04ada982bef50b4f80e961ead5fbcedae29adc211022"}
Sep 10 16:11:50.109515 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.109192    2572 scope.go:117] "RemoveContainer" containerID="24c4fed295eeaaf803cf0e707775e2eeb1063000988b025e2046f7bdd021826a"
Sep 10 16:11:50.168371 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.168337    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:11:50.264280 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.264188    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-istio-data\") pod \"111222db-994b-41f5-a88d-56876c84d47e\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") "
Sep 10 16:11:50.264280 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.264238    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-workload-certs\") pod \"111222db-994b-41f5-a88d-56876c84d47e\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") "
Sep 10 16:11:50.264280 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.264269    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111222db-994b-41f5-a88d-56876c84d47e-istio-podinfo\") pod \"111222db-994b-41f5-a88d-56876c84d47e\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") "
Sep 10 16:11:50.264683 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.264297    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-workload-socket\") pod \"111222db-994b-41f5-a88d-56876c84d47e\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") "
Sep 10 16:11:50.264683 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.264355    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111222db-994b-41f5-a88d-56876c84d47e-istiod-ca-cert\") pod \"111222db-994b-41f5-a88d-56876c84d47e\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") "
Sep 10 16:11:50.264683 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.264396    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-credential-socket\") pod \"111222db-994b-41f5-a88d-56876c84d47e\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") "
Sep 10 16:11:50.264683 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.264440    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbkzv\" (UniqueName: \"kubernetes.io/projected/111222db-994b-41f5-a88d-56876c84d47e-kube-api-access-qbkzv\") pod \"111222db-994b-41f5-a88d-56876c84d47e\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") "
Sep 10 16:11:50.264683 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.264472    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111222db-994b-41f5-a88d-56876c84d47e-istio-token\") pod \"111222db-994b-41f5-a88d-56876c84d47e\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") "
Sep 10 16:11:50.264683 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.264499    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-istio-envoy\") pod \"111222db-994b-41f5-a88d-56876c84d47e\" (UID: \"111222db-994b-41f5-a88d-56876c84d47e\") "
Sep 10 16:11:50.264683 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.264524    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-istio-data" (OuterVolumeSpecName: "istio-data") pod "111222db-994b-41f5-a88d-56876c84d47e" (UID: "111222db-994b-41f5-a88d-56876c84d47e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:11:50.264683 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.264573    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "111222db-994b-41f5-a88d-56876c84d47e" (UID: "111222db-994b-41f5-a88d-56876c84d47e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:11:50.265100 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.264729    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "111222db-994b-41f5-a88d-56876c84d47e" (UID: "111222db-994b-41f5-a88d-56876c84d47e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:11:50.265100 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.264788    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-istio-data\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:11:50.265100 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.264809    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-workload-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:11:50.265100 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.264823    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-workload-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:11:50.265100 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.264875    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "111222db-994b-41f5-a88d-56876c84d47e" (UID: "111222db-994b-41f5-a88d-56876c84d47e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:11:50.265100 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.265046    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/111222db-994b-41f5-a88d-56876c84d47e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "111222db-994b-41f5-a88d-56876c84d47e" (UID: "111222db-994b-41f5-a88d-56876c84d47e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:11:50.266688 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.266659    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111222db-994b-41f5-a88d-56876c84d47e-istio-token" (OuterVolumeSpecName: "istio-token") pod "111222db-994b-41f5-a88d-56876c84d47e" (UID: "111222db-994b-41f5-a88d-56876c84d47e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:11:50.266688 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.266678    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/111222db-994b-41f5-a88d-56876c84d47e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "111222db-994b-41f5-a88d-56876c84d47e" (UID: "111222db-994b-41f5-a88d-56876c84d47e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:11:50.266859 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.266746    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "111222db-994b-41f5-a88d-56876c84d47e" (UID: "111222db-994b-41f5-a88d-56876c84d47e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:11:50.266934 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.266905    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111222db-994b-41f5-a88d-56876c84d47e-kube-api-access-qbkzv" (OuterVolumeSpecName: "kube-api-access-qbkzv") pod "111222db-994b-41f5-a88d-56876c84d47e" (UID: "111222db-994b-41f5-a88d-56876c84d47e"). InnerVolumeSpecName "kube-api-access-qbkzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:11:50.365815 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.365776    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111222db-994b-41f5-a88d-56876c84d47e-istio-podinfo\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:11:50.365815 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.365806    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111222db-994b-41f5-a88d-56876c84d47e-istiod-ca-cert\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:11:50.365815 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.365815    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-credential-socket\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:11:50.365815 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.365825    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbkzv\" (UniqueName: \"kubernetes.io/projected/111222db-994b-41f5-a88d-56876c84d47e-kube-api-access-qbkzv\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:11:50.366101 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.365834    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111222db-994b-41f5-a88d-56876c84d47e-istio-token\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:11:50.366101 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:50.365842    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111222db-994b-41f5-a88d-56876c84d47e-istio-envoy\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:11:51.116216 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:51.116171    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt" event={"ID":"111222db-994b-41f5-a88d-56876c84d47e","Type":"ContainerDied","Data":"35538465c7b95d73ea69d7b831649ab110687d19c37b41fbed09ca145717917e"}
Sep 10 16:11:51.116709 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:51.116241    2572 scope.go:117] "RemoveContainer" containerID="09afb2dcd3dcfe607a3c04ada982bef50b4f80e961ead5fbcedae29adc211022"
Sep 10 16:11:51.116709 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:51.116186    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"
Sep 10 16:11:51.141247 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:51.141215    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"]
Sep 10 16:11:51.143339 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:51.143315    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-67df2b-openshift-default-5cd879b54d-r5szt"]
Sep 10 16:11:51.992808 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:11:51.992773    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="111222db-994b-41f5-a88d-56876c84d47e" path="/var/lib/kubelet/pods/111222db-994b-41f5-a88d-56876c84d47e/volumes"
Sep 10 16:12:07.972868 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:12:07.972835    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7sm72_8ddd4dc1-1122-41e0-a54a-67fb8008d7b9/ovn-acl-logging/0.log"
Sep 10 16:12:07.981838 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:12:07.981810    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7sm72_8ddd4dc1-1122-41e0-a54a-67fb8008d7b9/ovn-acl-logging/0.log"
Sep 10 16:13:08.434077 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:13:08.434039    2572 scope.go:117] "RemoveContainer" containerID="b597a41a9cbb925c474dc78eb126b4753959512854774768653c5dfd747a762f"
Sep 10 16:14:08.456204 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:08.456167    2572 scope.go:117] "RemoveContainer" containerID="fcc79c8f6c28ab650ee370246afaea054d256100a72d3f4e743883988ed0463b"
Sep 10 16:14:08.465665 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:08.465634    2572 scope.go:117] "RemoveContainer" containerID="aba58fb003cc8d5d42a5212dc3ab3da3a4b4f321c688e5b353194ffd0a8286c2"
Sep 10 16:14:08.475582 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:08.475555    2572 scope.go:117] "RemoveContainer" containerID="8a7d8f4754847aef62651e6ba59f51f439049d89a186d1d6d3ff191024563659"
Sep 10 16:14:31.966565 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:31.966521    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k"]
Sep 10 16:14:31.967136 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:31.966725    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k" podUID="c3a4eb26-1798-4e4e-a16e-d76c441bf883" containerName="manager" containerID="cri-o://e851d5250d0d2acba7afdae5c5123c2835fe9126c48e2e48edaae1887c25cdd7" gracePeriod=10
Sep 10 16:14:32.225011 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:32.224937    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k"
Sep 10 16:14:32.406486 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:32.406445    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vflj2\" (UniqueName: \"kubernetes.io/projected/c3a4eb26-1798-4e4e-a16e-d76c441bf883-kube-api-access-vflj2\") pod \"c3a4eb26-1798-4e4e-a16e-d76c441bf883\" (UID: \"c3a4eb26-1798-4e4e-a16e-d76c441bf883\") "
Sep 10 16:14:32.406683 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:32.406540    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3a4eb26-1798-4e4e-a16e-d76c441bf883-extensions-socket-volume\") pod \"c3a4eb26-1798-4e4e-a16e-d76c441bf883\" (UID: \"c3a4eb26-1798-4e4e-a16e-d76c441bf883\") "
Sep 10 16:14:32.406960 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:32.406936    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a4eb26-1798-4e4e-a16e-d76c441bf883-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c3a4eb26-1798-4e4e-a16e-d76c441bf883" (UID: "c3a4eb26-1798-4e4e-a16e-d76c441bf883"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:14:32.408743 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:32.408717    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a4eb26-1798-4e4e-a16e-d76c441bf883-kube-api-access-vflj2" (OuterVolumeSpecName: "kube-api-access-vflj2") pod "c3a4eb26-1798-4e4e-a16e-d76c441bf883" (UID: "c3a4eb26-1798-4e4e-a16e-d76c441bf883"). InnerVolumeSpecName "kube-api-access-vflj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:14:32.507631 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:32.507536    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vflj2\" (UniqueName: \"kubernetes.io/projected/c3a4eb26-1798-4e4e-a16e-d76c441bf883-kube-api-access-vflj2\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:14:32.507631 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:32.507570    2572 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3a4eb26-1798-4e4e-a16e-d76c441bf883-extensions-socket-volume\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:14:33.030481 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:33.030441    2572 generic.go:356] "Generic (PLEG): container finished" podID="c3a4eb26-1798-4e4e-a16e-d76c441bf883" containerID="e851d5250d0d2acba7afdae5c5123c2835fe9126c48e2e48edaae1887c25cdd7" exitCode=0
Sep 10 16:14:33.030911 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:33.030496    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k"
Sep 10 16:14:33.030911 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:33.030528    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k" event={"ID":"c3a4eb26-1798-4e4e-a16e-d76c441bf883","Type":"ContainerDied","Data":"e851d5250d0d2acba7afdae5c5123c2835fe9126c48e2e48edaae1887c25cdd7"}
Sep 10 16:14:33.030911 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:33.030585    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k" event={"ID":"c3a4eb26-1798-4e4e-a16e-d76c441bf883","Type":"ContainerDied","Data":"2f9d11f2017d5a7c302d56a1978a932e61345da843e70ff3dca5f100c6411e81"}
Sep 10 16:14:33.030911 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:33.030608    2572 scope.go:117] "RemoveContainer" containerID="e851d5250d0d2acba7afdae5c5123c2835fe9126c48e2e48edaae1887c25cdd7"
Sep 10 16:14:33.040679 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:33.040651    2572 scope.go:117] "RemoveContainer" containerID="e851d5250d0d2acba7afdae5c5123c2835fe9126c48e2e48edaae1887c25cdd7"
Sep 10 16:14:33.040990 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:14:33.040969    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e851d5250d0d2acba7afdae5c5123c2835fe9126c48e2e48edaae1887c25cdd7\": container with ID starting with e851d5250d0d2acba7afdae5c5123c2835fe9126c48e2e48edaae1887c25cdd7 not found: ID does not exist" containerID="e851d5250d0d2acba7afdae5c5123c2835fe9126c48e2e48edaae1887c25cdd7"
Sep 10 16:14:33.041051 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:33.041000    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e851d5250d0d2acba7afdae5c5123c2835fe9126c48e2e48edaae1887c25cdd7"} err="failed to get container status \"e851d5250d0d2acba7afdae5c5123c2835fe9126c48e2e48edaae1887c25cdd7\": rpc error: code = NotFound desc = could not find container \"e851d5250d0d2acba7afdae5c5123c2835fe9126c48e2e48edaae1887c25cdd7\": container with ID starting with e851d5250d0d2acba7afdae5c5123c2835fe9126c48e2e48edaae1887c25cdd7 not found: ID does not exist"
Sep 10 16:14:33.054000 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:33.053962    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k"]
Sep 10 16:14:33.058795 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:33.058764    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-p428k"]
Sep 10 16:14:33.992147 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:14:33.992113    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3a4eb26-1798-4e4e-a16e-d76c441bf883" path="/var/lib/kubelet/pods/c3a4eb26-1798-4e4e-a16e-d76c441bf883/volumes"
Sep 10 16:15:32.640482 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.640449    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fp8hw"]
Sep 10 16:15:32.640915 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.640810    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="111222db-994b-41f5-a88d-56876c84d47e" containerName="istio-proxy"
Sep 10 16:15:32.640915 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.640823    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="111222db-994b-41f5-a88d-56876c84d47e" containerName="istio-proxy"
Sep 10 16:15:32.640915 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.640831    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="111222db-994b-41f5-a88d-56876c84d47e" containerName="istio-proxy"
Sep 10 16:15:32.640915 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.640836    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="111222db-994b-41f5-a88d-56876c84d47e" containerName="istio-proxy"
Sep 10 16:15:32.640915 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.640844    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51c87e36-1cc9-4711-8220-e379dbd36f30" containerName="istio-proxy"
Sep 10 16:15:32.640915 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.640849    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="51c87e36-1cc9-4711-8220-e379dbd36f30" containerName="istio-proxy"
Sep 10 16:15:32.640915 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.640857    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3a4eb26-1798-4e4e-a16e-d76c441bf883" containerName="manager"
Sep 10 16:15:32.640915 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.640862    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3a4eb26-1798-4e4e-a16e-d76c441bf883" containerName="manager"
Sep 10 16:15:32.640915 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.640872    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8606646b-7554-44de-9672-e9c8182782fa" containerName="revoke-keys"
Sep 10 16:15:32.640915 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.640877    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8606646b-7554-44de-9672-e9c8182782fa" containerName="revoke-keys"
Sep 10 16:15:32.640915 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.640883    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51c87e36-1cc9-4711-8220-e379dbd36f30" containerName="istio-proxy"
Sep 10 16:15:32.640915 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.640888    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="51c87e36-1cc9-4711-8220-e379dbd36f30" containerName="istio-proxy"
Sep 10 16:15:32.641456 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.640939    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="111222db-994b-41f5-a88d-56876c84d47e" containerName="istio-proxy"
Sep 10 16:15:32.641456 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.640945    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="111222db-994b-41f5-a88d-56876c84d47e" containerName="istio-proxy"
Sep 10 16:15:32.641456 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.640951    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8606646b-7554-44de-9672-e9c8182782fa" containerName="revoke-keys"
Sep 10 16:15:32.641456 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.640958    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3a4eb26-1798-4e4e-a16e-d76c441bf883" containerName="manager"
Sep 10 16:15:32.641456 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.640964    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="51c87e36-1cc9-4711-8220-e379dbd36f30" containerName="istio-proxy"
Sep 10 16:15:32.641456 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.640969    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="51c87e36-1cc9-4711-8220-e379dbd36f30" containerName="istio-proxy"
Sep 10 16:15:32.643941 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.643920    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fp8hw"
Sep 10 16:15:32.646092 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.646069    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5ls97\""
Sep 10 16:15:32.656358 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.656330    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fp8hw"]
Sep 10 16:15:32.742891 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.742835    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr92g\" (UniqueName: \"kubernetes.io/projected/0d8c5820-0773-4326-a315-c44c321b789d-kube-api-access-kr92g\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fp8hw\" (UID: \"0d8c5820-0773-4326-a315-c44c321b789d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fp8hw"
Sep 10 16:15:32.742891 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.742892    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d8c5820-0773-4326-a315-c44c321b789d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fp8hw\" (UID: \"0d8c5820-0773-4326-a315-c44c321b789d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fp8hw"
Sep 10 16:15:32.843509 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.843457    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr92g\" (UniqueName: \"kubernetes.io/projected/0d8c5820-0773-4326-a315-c44c321b789d-kube-api-access-kr92g\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fp8hw\" (UID: \"0d8c5820-0773-4326-a315-c44c321b789d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fp8hw"
Sep 10 16:15:32.843509 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.843513    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d8c5820-0773-4326-a315-c44c321b789d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fp8hw\" (UID: \"0d8c5820-0773-4326-a315-c44c321b789d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fp8hw"
Sep 10 16:15:32.843890 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.843871    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d8c5820-0773-4326-a315-c44c321b789d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fp8hw\" (UID: \"0d8c5820-0773-4326-a315-c44c321b789d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fp8hw"
Sep 10 16:15:32.855463 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.855440    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr92g\" (UniqueName: \"kubernetes.io/projected/0d8c5820-0773-4326-a315-c44c321b789d-kube-api-access-kr92g\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fp8hw\" (UID: \"0d8c5820-0773-4326-a315-c44c321b789d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fp8hw"
Sep 10 16:15:32.954775 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:32.954665    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fp8hw"
Sep 10 16:15:33.235638 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:33.235600    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fp8hw"]
Sep 10 16:15:33.237834 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:15:33.237810    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d8c5820_0773_4326_a315_c44c321b789d.slice/crio-28fce2ffe256df5dd3401c5f980f6c159aa1add1c0be4ce7f557dc560ccb4f40 WatchSource:0}: Error finding container 28fce2ffe256df5dd3401c5f980f6c159aa1add1c0be4ce7f557dc560ccb4f40: Status 404 returned error can't find the container with id 28fce2ffe256df5dd3401c5f980f6c159aa1add1c0be4ce7f557dc560ccb4f40
Sep 10 16:15:33.343810 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:33.343770    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fp8hw" event={"ID":"0d8c5820-0773-4326-a315-c44c321b789d","Type":"ContainerStarted","Data":"3e6dd7db1cddf883f4d6be5a50c59f513d22046d5744ccd8cce9bc44d3a8fd4f"}
Sep 10 16:15:33.344002 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:33.343921    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fp8hw"
Sep 10 16:15:33.344002 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:33.343963    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fp8hw" event={"ID":"0d8c5820-0773-4326-a315-c44c321b789d","Type":"ContainerStarted","Data":"28fce2ffe256df5dd3401c5f980f6c159aa1add1c0be4ce7f557dc560ccb4f40"}
Sep 10 16:15:33.368541 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:33.368480    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fp8hw" podStartSLOduration=1.36846423 podStartE2EDuration="1.36846423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:15:33.367132713 +0000 UTC m=+2006.039542796" watchObservedRunningTime="2026-09-10 16:15:33.36846423 +0000 UTC m=+2006.040874313"
Sep 10 16:15:44.351726 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:15:44.351688    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fp8hw"
Sep 10 16:16:06.256372 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.256324    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"]
Sep 10 16:16:06.260291 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.260269    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.262661 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.262633    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-bf5277c2-kserve-self-signed-certs\""
Sep 10 16:16:06.267852 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.267521    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"]
Sep 10 16:16:06.355582 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.355531    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-tmp-dir\") pod \"e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") " pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.355582 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.355578    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-tls-certs\") pod \"e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") " pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.355818 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.355649    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xjsp\" (UniqueName: \"kubernetes.io/projected/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-kube-api-access-2xjsp\") pod \"e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") " pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.355818 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.355681    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-model-cache\") pod \"e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") " pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.355818 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.355711    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-dshm\") pod \"e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") " pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.355818 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.355810    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-home\") pod \"e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") " pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.456525 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.456476    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-home\") pod \"e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") " pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.456720 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.456542    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-tmp-dir\") pod \"e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") " pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.456720 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.456558    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-tls-certs\") pod \"e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") " pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.456720 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.456585    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjsp\" (UniqueName: \"kubernetes.io/projected/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-kube-api-access-2xjsp\") pod \"e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") " pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.456720 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.456603    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-model-cache\") pod \"e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") " pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.456720 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.456627    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-dshm\") pod \"e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") " pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.457006 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.456966    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-home\") pod \"e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") " pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.457131 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.456977    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-tmp-dir\") pod \"e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") " pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.457131 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.457029    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-model-cache\") pod \"e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") " pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.458985 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.458961    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-dshm\") pod \"e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") " pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.459236 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.459219    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-tls-certs\") pod \"e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") " pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.465451 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.465423    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xjsp\" (UniqueName: \"kubernetes.io/projected/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-kube-api-access-2xjsp\") pod \"e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") " pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.571468 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.571431    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:06.844470 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:06.844270    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"]
Sep 10 16:16:06.847380 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:16:06.847349    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15f9c6b0_18f5_4b99_abf1_be34c989f9ff.slice/crio-f9d4b14d9642031224abfd6f3b52f40cf4f45da08f83a532248237e0ceddbaed WatchSource:0}: Error finding container f9d4b14d9642031224abfd6f3b52f40cf4f45da08f83a532248237e0ceddbaed: Status 404 returned error can't find the container with id f9d4b14d9642031224abfd6f3b52f40cf4f45da08f83a532248237e0ceddbaed
Sep 10 16:16:07.530410 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:07.530367    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm" event={"ID":"15f9c6b0-18f5-4b99-abf1-be34c989f9ff","Type":"ContainerStarted","Data":"43341659f4b786dc7ed11cbd557bb802ec67a0b36f0937aaa516ade76734a2fd"}
Sep 10 16:16:07.530410 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:07.530415    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm" event={"ID":"15f9c6b0-18f5-4b99-abf1-be34c989f9ff","Type":"ContainerStarted","Data":"f9d4b14d9642031224abfd6f3b52f40cf4f45da08f83a532248237e0ceddbaed"}
Sep 10 16:16:07.547592 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:07.547511    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm" podStartSLOduration=1.547495711 podStartE2EDuration="1.547495711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:16:07.546738681 +0000 UTC m=+2040.219148764" watchObservedRunningTime="2026-09-10 16:16:07.547495711 +0000 UTC m=+2040.219905793"
Sep 10 16:16:08.531726 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:08.531687    2572 scope.go:117] "RemoveContainer" containerID="fffbe0ac7e75ed2574520905d1508d5d05b8ab068aac5990c6ec82aced36e1f7"
Sep 10 16:16:10.646551 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.646511    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"]
Sep 10 16:16:10.651578 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.651556    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:10.653921 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.653894    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-bf5277c2-kserve-self-signed-certs\""
Sep 10 16:16:10.657676 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.657650    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"]
Sep 10 16:16:10.694275 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.694232    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-model-cache\") pod \"e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") " pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:10.694275 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.694278    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-tmp-dir\") pod \"e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") " pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:10.694544 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.694387    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk4m7\" (UniqueName: \"kubernetes.io/projected/28db2447-b5c0-466e-b679-f3cc23098531-kube-api-access-nk4m7\") pod \"e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") " pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:10.694544 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.694427    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-dshm\") pod \"e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") " pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:10.694544 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.694448    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-home\") pod \"e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") " pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:10.694544 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.694472    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28db2447-b5c0-466e-b679-f3cc23098531-tls-certs\") pod \"e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") " pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:10.795650 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.795613    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-tmp-dir\") pod \"e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") " pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:10.795854 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.795668    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk4m7\" (UniqueName: \"kubernetes.io/projected/28db2447-b5c0-466e-b679-f3cc23098531-kube-api-access-nk4m7\") pod \"e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") " pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:10.795854 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.795692    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-dshm\") pod \"e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") " pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:10.795854 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.795714    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-home\") pod \"e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") " pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:10.795854 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.795745    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28db2447-b5c0-466e-b679-f3cc23098531-tls-certs\") pod \"e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") " pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:10.795854 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.795802    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-model-cache\") pod \"e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") " pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:10.796160 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.796100    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-tmp-dir\") pod \"e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") " pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:10.796160 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.796124    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-model-cache\") pod \"e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") " pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:10.796233 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.796178    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-home\") pod \"e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") " pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:10.798111 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.798088    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-dshm\") pod \"e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") " pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:10.798274 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.798255    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28db2447-b5c0-466e-b679-f3cc23098531-tls-certs\") pod \"e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") " pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:10.803215 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.803193    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk4m7\" (UniqueName: \"kubernetes.io/projected/28db2447-b5c0-466e-b679-f3cc23098531-kube-api-access-nk4m7\") pod \"e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") " pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:10.961466 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:10.961366    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:11.241108 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:11.241075    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"]
Sep 10 16:16:11.242464 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:16:11.242435    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28db2447_b5c0_466e_b679_f3cc23098531.slice/crio-7a66a9f63df268882af855c7bebdb1c3d5c2ca343ea779e99417cfb3f57af18b WatchSource:0}: Error finding container 7a66a9f63df268882af855c7bebdb1c3d5c2ca343ea779e99417cfb3f57af18b: Status 404 returned error can't find the container with id 7a66a9f63df268882af855c7bebdb1c3d5c2ca343ea779e99417cfb3f57af18b
Sep 10 16:16:11.555493 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:11.555453    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc" event={"ID":"28db2447-b5c0-466e-b679-f3cc23098531","Type":"ContainerStarted","Data":"a4b492a81f9f83e0b63effd70b13dc5addcd18979e638ca4729eb73e8d4ba5f2"}
Sep 10 16:16:11.555679 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:11.555509    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc" event={"ID":"28db2447-b5c0-466e-b679-f3cc23098531","Type":"ContainerStarted","Data":"7a66a9f63df268882af855c7bebdb1c3d5c2ca343ea779e99417cfb3f57af18b"}
Sep 10 16:16:11.574537 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:11.574477    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc" podStartSLOduration=1.5744560459999999 podStartE2EDuration="1.574456046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:16:11.571907803 +0000 UTC m=+2044.244317885" watchObservedRunningTime="2026-09-10 16:16:11.574456046 +0000 UTC m=+2044.246866128"
Sep 10 16:16:16.572545 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:16.572472    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:16.572996 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:16.572579    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:16.585906 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:16.585865    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:16.597399 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:16.597368    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:20.962113 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:20.962075    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:20.962113 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:20.962125    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:20.974647 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:20.974618    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:21.628393 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:21.628355    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:40.059147 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:40.059044    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"]
Sep 10 16:16:41.497135 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:41.497100    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"]
Sep 10 16:16:55.079261 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.079216    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm" podUID="15f9c6b0-18f5-4b99-abf1-be34c989f9ff" containerName="main" containerID="cri-o://43341659f4b786dc7ed11cbd557bb802ec67a0b36f0937aaa516ade76734a2fd" gracePeriod=45
Sep 10 16:16:55.331722 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.331651    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:55.414282 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.414241    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-model-cache\") pod \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") "
Sep 10 16:16:55.414282 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.414285    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xjsp\" (UniqueName: \"kubernetes.io/projected/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-kube-api-access-2xjsp\") pod \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") "
Sep 10 16:16:55.414566 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.414330    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-tmp-dir\") pod \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") "
Sep 10 16:16:55.414566 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.414382    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-home\") pod \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") "
Sep 10 16:16:55.414566 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.414426    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-tls-certs\") pod \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") "
Sep 10 16:16:55.414566 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.414451    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-dshm\") pod \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\" (UID: \"15f9c6b0-18f5-4b99-abf1-be34c989f9ff\") "
Sep 10 16:16:55.414791 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.414627    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "15f9c6b0-18f5-4b99-abf1-be34c989f9ff" (UID: "15f9c6b0-18f5-4b99-abf1-be34c989f9ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:16:55.414791 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.414687    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-home" (OuterVolumeSpecName: "home") pod "15f9c6b0-18f5-4b99-abf1-be34c989f9ff" (UID: "15f9c6b0-18f5-4b99-abf1-be34c989f9ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:16:55.414791 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.414698    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "15f9c6b0-18f5-4b99-abf1-be34c989f9ff" (UID: "15f9c6b0-18f5-4b99-abf1-be34c989f9ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:16:55.416579 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.416542    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "15f9c6b0-18f5-4b99-abf1-be34c989f9ff" (UID: "15f9c6b0-18f5-4b99-abf1-be34c989f9ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:16:55.416696 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.416552    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-dshm" (OuterVolumeSpecName: "dshm") pod "15f9c6b0-18f5-4b99-abf1-be34c989f9ff" (UID: "15f9c6b0-18f5-4b99-abf1-be34c989f9ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:16:55.416696 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.416587    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-kube-api-access-2xjsp" (OuterVolumeSpecName: "kube-api-access-2xjsp") pod "15f9c6b0-18f5-4b99-abf1-be34c989f9ff" (UID: "15f9c6b0-18f5-4b99-abf1-be34c989f9ff"). InnerVolumeSpecName "kube-api-access-2xjsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:16:55.515804 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.515762    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-home\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:16:55.515804 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.515795    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-tls-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:16:55.515804 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.515804    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-dshm\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:16:55.515804 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.515814    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-model-cache\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:16:55.516087 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.515822    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xjsp\" (UniqueName: \"kubernetes.io/projected/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-kube-api-access-2xjsp\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:16:55.516087 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.515830    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15f9c6b0-18f5-4b99-abf1-be34c989f9ff-tmp-dir\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:16:55.807457 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.807413    2572 generic.go:356] "Generic (PLEG): container finished" podID="15f9c6b0-18f5-4b99-abf1-be34c989f9ff" containerID="43341659f4b786dc7ed11cbd557bb802ec67a0b36f0937aaa516ade76734a2fd" exitCode=0
Sep 10 16:16:55.807457 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.807456    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"
Sep 10 16:16:55.807697 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.807496    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm" event={"ID":"15f9c6b0-18f5-4b99-abf1-be34c989f9ff","Type":"ContainerDied","Data":"43341659f4b786dc7ed11cbd557bb802ec67a0b36f0937aaa516ade76734a2fd"}
Sep 10 16:16:55.807697 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.807533    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm" event={"ID":"15f9c6b0-18f5-4b99-abf1-be34c989f9ff","Type":"ContainerDied","Data":"f9d4b14d9642031224abfd6f3b52f40cf4f45da08f83a532248237e0ceddbaed"}
Sep 10 16:16:55.807697 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.807550    2572 scope.go:117] "RemoveContainer" containerID="43341659f4b786dc7ed11cbd557bb802ec67a0b36f0937aaa516ade76734a2fd"
Sep 10 16:16:55.817718 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.817637    2572 scope.go:117] "RemoveContainer" containerID="43341659f4b786dc7ed11cbd557bb802ec67a0b36f0937aaa516ade76734a2fd"
Sep 10 16:16:55.817941 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:16:55.817921    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43341659f4b786dc7ed11cbd557bb802ec67a0b36f0937aaa516ade76734a2fd\": container with ID starting with 43341659f4b786dc7ed11cbd557bb802ec67a0b36f0937aaa516ade76734a2fd not found: ID does not exist" containerID="43341659f4b786dc7ed11cbd557bb802ec67a0b36f0937aaa516ade76734a2fd"
Sep 10 16:16:55.818013 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.817950    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43341659f4b786dc7ed11cbd557bb802ec67a0b36f0937aaa516ade76734a2fd"} err="failed to get container status \"43341659f4b786dc7ed11cbd557bb802ec67a0b36f0937aaa516ade76734a2fd\": rpc error: code = NotFound desc = could not find container \"43341659f4b786dc7ed11cbd557bb802ec67a0b36f0937aaa516ade76734a2fd\": container with ID starting with 43341659f4b786dc7ed11cbd557bb802ec67a0b36f0937aaa516ade76734a2fd not found: ID does not exist"
Sep 10 16:16:55.828674 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.828642    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"]
Sep 10 16:16:55.833204 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.833173    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-bf5277c2-kserve-5bb586979f-cplqm"]
Sep 10 16:16:55.992513 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:55.992475    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15f9c6b0-18f5-4b99-abf1-be34c989f9ff" path="/var/lib/kubelet/pods/15f9c6b0-18f5-4b99-abf1-be34c989f9ff/volumes"
Sep 10 16:16:56.516861 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.516795    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc" podUID="28db2447-b5c0-466e-b679-f3cc23098531" containerName="main" containerID="cri-o://a4b492a81f9f83e0b63effd70b13dc5addcd18979e638ca4729eb73e8d4ba5f2" gracePeriod=45
Sep 10 16:16:56.761095 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.761068    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:56.815863 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.815825    2572 generic.go:356] "Generic (PLEG): container finished" podID="28db2447-b5c0-466e-b679-f3cc23098531" containerID="a4b492a81f9f83e0b63effd70b13dc5addcd18979e638ca4729eb73e8d4ba5f2" exitCode=0
Sep 10 16:16:56.816052 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.815866    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"
Sep 10 16:16:56.816052 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.815885    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc" event={"ID":"28db2447-b5c0-466e-b679-f3cc23098531","Type":"ContainerDied","Data":"a4b492a81f9f83e0b63effd70b13dc5addcd18979e638ca4729eb73e8d4ba5f2"}
Sep 10 16:16:56.816052 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.815921    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc" event={"ID":"28db2447-b5c0-466e-b679-f3cc23098531","Type":"ContainerDied","Data":"7a66a9f63df268882af855c7bebdb1c3d5c2ca343ea779e99417cfb3f57af18b"}
Sep 10 16:16:56.816052 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.815941    2572 scope.go:117] "RemoveContainer" containerID="a4b492a81f9f83e0b63effd70b13dc5addcd18979e638ca4729eb73e8d4ba5f2"
Sep 10 16:16:56.828481 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.828451    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-dshm\") pod \"28db2447-b5c0-466e-b679-f3cc23098531\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") "
Sep 10 16:16:56.828631 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.828551    2572 scope.go:117] "RemoveContainer" containerID="a4b492a81f9f83e0b63effd70b13dc5addcd18979e638ca4729eb73e8d4ba5f2"
Sep 10 16:16:56.828631 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.828550    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-tmp-dir\") pod \"28db2447-b5c0-466e-b679-f3cc23098531\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") "
Sep 10 16:16:56.828756 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.828729    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-model-cache\") pod \"28db2447-b5c0-466e-b679-f3cc23098531\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") "
Sep 10 16:16:56.828827 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.828779    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28db2447-b5c0-466e-b679-f3cc23098531-tls-certs\") pod \"28db2447-b5c0-466e-b679-f3cc23098531\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") "
Sep 10 16:16:56.828827 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.828801    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "28db2447-b5c0-466e-b679-f3cc23098531" (UID: "28db2447-b5c0-466e-b679-f3cc23098531"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:16:56.828827 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.828820    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nk4m7\" (UniqueName: \"kubernetes.io/projected/28db2447-b5c0-466e-b679-f3cc23098531-kube-api-access-nk4m7\") pod \"28db2447-b5c0-466e-b679-f3cc23098531\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") "
Sep 10 16:16:56.828982 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.828869    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-home\") pod \"28db2447-b5c0-466e-b679-f3cc23098531\" (UID: \"28db2447-b5c0-466e-b679-f3cc23098531\") "
Sep 10 16:16:56.828982 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.828971    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-model-cache" (OuterVolumeSpecName: "model-cache") pod "28db2447-b5c0-466e-b679-f3cc23098531" (UID: "28db2447-b5c0-466e-b679-f3cc23098531"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:16:56.829099 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:16:56.829064    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4b492a81f9f83e0b63effd70b13dc5addcd18979e638ca4729eb73e8d4ba5f2\": container with ID starting with a4b492a81f9f83e0b63effd70b13dc5addcd18979e638ca4729eb73e8d4ba5f2 not found: ID does not exist" containerID="a4b492a81f9f83e0b63effd70b13dc5addcd18979e638ca4729eb73e8d4ba5f2"
Sep 10 16:16:56.829172 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.829101    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b492a81f9f83e0b63effd70b13dc5addcd18979e638ca4729eb73e8d4ba5f2"} err="failed to get container status \"a4b492a81f9f83e0b63effd70b13dc5addcd18979e638ca4729eb73e8d4ba5f2\": rpc error: code = NotFound desc = could not find container \"a4b492a81f9f83e0b63effd70b13dc5addcd18979e638ca4729eb73e8d4ba5f2\": container with ID starting with a4b492a81f9f83e0b63effd70b13dc5addcd18979e638ca4729eb73e8d4ba5f2 not found: ID does not exist"
Sep 10 16:16:56.829172 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.829158    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-tmp-dir\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:16:56.829282 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.829173    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-model-cache\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:16:56.829449 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.829368    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-home" (OuterVolumeSpecName: "home") pod "28db2447-b5c0-466e-b679-f3cc23098531" (UID: "28db2447-b5c0-466e-b679-f3cc23098531"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:16:56.830817 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.830788    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-dshm" (OuterVolumeSpecName: "dshm") pod "28db2447-b5c0-466e-b679-f3cc23098531" (UID: "28db2447-b5c0-466e-b679-f3cc23098531"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:16:56.831031 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.831014    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28db2447-b5c0-466e-b679-f3cc23098531-kube-api-access-nk4m7" (OuterVolumeSpecName: "kube-api-access-nk4m7") pod "28db2447-b5c0-466e-b679-f3cc23098531" (UID: "28db2447-b5c0-466e-b679-f3cc23098531"). InnerVolumeSpecName "kube-api-access-nk4m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:16:56.831221 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.831205    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28db2447-b5c0-466e-b679-f3cc23098531-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "28db2447-b5c0-466e-b679-f3cc23098531" (UID: "28db2447-b5c0-466e-b679-f3cc23098531"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:16:56.930179 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.930137    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-dshm\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:16:56.930179 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.930176    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28db2447-b5c0-466e-b679-f3cc23098531-tls-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:16:56.930412 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.930192    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk4m7\" (UniqueName: \"kubernetes.io/projected/28db2447-b5c0-466e-b679-f3cc23098531-kube-api-access-nk4m7\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:16:56.930412 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:56.930206    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28db2447-b5c0-466e-b679-f3cc23098531-home\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:16:57.141927 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:57.141888    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"]
Sep 10 16:16:57.146402 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:57.146372    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-bf5277c2-kserve-5878bff6f9-t54pc"]
Sep 10 16:16:57.993145 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:57.993105    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28db2447-b5c0-466e-b679-f3cc23098531" path="/var/lib/kubelet/pods/28db2447-b5c0-466e-b679-f3cc23098531/volumes"
Sep 10 16:16:59.803706 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.803664    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"]
Sep 10 16:16:59.804220 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.804201    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28db2447-b5c0-466e-b679-f3cc23098531" containerName="main"
Sep 10 16:16:59.804343 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.804223    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="28db2447-b5c0-466e-b679-f3cc23098531" containerName="main"
Sep 10 16:16:59.804343 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.804249    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15f9c6b0-18f5-4b99-abf1-be34c989f9ff" containerName="main"
Sep 10 16:16:59.804343 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.804259    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="15f9c6b0-18f5-4b99-abf1-be34c989f9ff" containerName="main"
Sep 10 16:16:59.804504 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.804406    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="28db2447-b5c0-466e-b679-f3cc23098531" containerName="main"
Sep 10 16:16:59.804504 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.804422    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="15f9c6b0-18f5-4b99-abf1-be34c989f9ff" containerName="main"
Sep 10 16:16:59.808973 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.808949    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"
Sep 10 16:16:59.811523 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.811499    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 16:16:59.812374 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.812349    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:16:59.812485 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.812354    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 16:16:59.812561 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.812483    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nwclg\""
Sep 10 16:16:59.812561 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.812525    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 16:16:59.823349 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.823296    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"]
Sep 10 16:16:59.857398 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.857366    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6948d5ca-0320-44d5-ac2a-17e8968b059b-openshift-service-ca\") pod \"maas-api-6689875f55-4x9kk\" (UID: \"6948d5ca-0320-44d5-ac2a-17e8968b059b\") " pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"
Sep 10 16:16:59.857581 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.857415    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf4tc\" (UniqueName: \"kubernetes.io/projected/6948d5ca-0320-44d5-ac2a-17e8968b059b-kube-api-access-lf4tc\") pod \"maas-api-6689875f55-4x9kk\" (UID: \"6948d5ca-0320-44d5-ac2a-17e8968b059b\") " pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"
Sep 10 16:16:59.857581 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.857525    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6948d5ca-0320-44d5-ac2a-17e8968b059b-metrics-certs\") pod \"maas-api-6689875f55-4x9kk\" (UID: \"6948d5ca-0320-44d5-ac2a-17e8968b059b\") " pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"
Sep 10 16:16:59.857581 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.857566    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6948d5ca-0320-44d5-ac2a-17e8968b059b-maas-api-tls\") pod \"maas-api-6689875f55-4x9kk\" (UID: \"6948d5ca-0320-44d5-ac2a-17e8968b059b\") " pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"
Sep 10 16:16:59.958658 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.958620    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6948d5ca-0320-44d5-ac2a-17e8968b059b-metrics-certs\") pod \"maas-api-6689875f55-4x9kk\" (UID: \"6948d5ca-0320-44d5-ac2a-17e8968b059b\") " pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"
Sep 10 16:16:59.958658 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.958661    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6948d5ca-0320-44d5-ac2a-17e8968b059b-maas-api-tls\") pod \"maas-api-6689875f55-4x9kk\" (UID: \"6948d5ca-0320-44d5-ac2a-17e8968b059b\") " pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"
Sep 10 16:16:59.958889 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.958726    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6948d5ca-0320-44d5-ac2a-17e8968b059b-openshift-service-ca\") pod \"maas-api-6689875f55-4x9kk\" (UID: \"6948d5ca-0320-44d5-ac2a-17e8968b059b\") " pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"
Sep 10 16:16:59.958889 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.958772    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf4tc\" (UniqueName: \"kubernetes.io/projected/6948d5ca-0320-44d5-ac2a-17e8968b059b-kube-api-access-lf4tc\") pod \"maas-api-6689875f55-4x9kk\" (UID: \"6948d5ca-0320-44d5-ac2a-17e8968b059b\") " pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"
Sep 10 16:16:59.959506 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.959485    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6948d5ca-0320-44d5-ac2a-17e8968b059b-openshift-service-ca\") pod \"maas-api-6689875f55-4x9kk\" (UID: \"6948d5ca-0320-44d5-ac2a-17e8968b059b\") " pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"
Sep 10 16:16:59.961248 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.961220    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6948d5ca-0320-44d5-ac2a-17e8968b059b-maas-api-tls\") pod \"maas-api-6689875f55-4x9kk\" (UID: \"6948d5ca-0320-44d5-ac2a-17e8968b059b\") " pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"
Sep 10 16:16:59.961428 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.961407    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6948d5ca-0320-44d5-ac2a-17e8968b059b-metrics-certs\") pod \"maas-api-6689875f55-4x9kk\" (UID: \"6948d5ca-0320-44d5-ac2a-17e8968b059b\") " pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"
Sep 10 16:16:59.966446 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:16:59.966427    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf4tc\" (UniqueName: \"kubernetes.io/projected/6948d5ca-0320-44d5-ac2a-17e8968b059b-kube-api-access-lf4tc\") pod \"maas-api-6689875f55-4x9kk\" (UID: \"6948d5ca-0320-44d5-ac2a-17e8968b059b\") " pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"
Sep 10 16:17:00.120405 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:00.120294    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"
Sep 10 16:17:00.401361 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:00.401331    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"]
Sep 10 16:17:00.403162 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:17:00.403133    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6948d5ca_0320_44d5_ac2a_17e8968b059b.slice/crio-c077774f401d9d446654ac58fd751cae3eaafe814055200a6e0765bc2c51d4f5 WatchSource:0}: Error finding container c077774f401d9d446654ac58fd751cae3eaafe814055200a6e0765bc2c51d4f5: Status 404 returned error can't find the container with id c077774f401d9d446654ac58fd751cae3eaafe814055200a6e0765bc2c51d4f5
Sep 10 16:17:00.404666 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:00.404645    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 16:17:00.845068 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:00.845030    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk" event={"ID":"6948d5ca-0320-44d5-ac2a-17e8968b059b","Type":"ContainerStarted","Data":"c077774f401d9d446654ac58fd751cae3eaafe814055200a6e0765bc2c51d4f5"}
Sep 10 16:17:01.851615 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:01.851574    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk" event={"ID":"6948d5ca-0320-44d5-ac2a-17e8968b059b","Type":"ContainerStarted","Data":"9a27c56d29776c21cefda13d383b1abad55e28c5f4b3b0470b645f553d136590"}
Sep 10 16:17:01.852030 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:01.851650    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"
Sep 10 16:17:01.869003 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:01.868947    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk" podStartSLOduration=2.403959995 podStartE2EDuration="2.868929733s" totalImagesPullingTime="464.969738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:16:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:01.868168219 +0000 UTC m=+2094.540578302" watchObservedRunningTime="2026-09-10 16:17:01.868929733 +0000 UTC m=+2094.541339865"
Sep 10 16:17:07.210521 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.210485    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"]
Sep 10 16:17:07.210940 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.210673    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk" podUID="6948d5ca-0320-44d5-ac2a-17e8968b059b" containerName="maas-api" containerID="cri-o://9a27c56d29776c21cefda13d383b1abad55e28c5f4b3b0470b645f553d136590" gracePeriod=30
Sep 10 16:17:07.216118 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.216094    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"
Sep 10 16:17:07.273925 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.273887    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl"]
Sep 10 16:17:07.277401 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.277377    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl"
Sep 10 16:17:07.291203 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.291166    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl"]
Sep 10 16:17:07.326762 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.326732    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed578f8-321b-446d-843c-b022ccc5e5ca-openshift-service-ca\") pod \"maas-api-79df8fbc7f-x4bsl\" (UID: \"1ed578f8-321b-446d-843c-b022ccc5e5ca\") " pod="odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl"
Sep 10 16:17:07.326920 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.326808    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ed578f8-321b-446d-843c-b022ccc5e5ca-metrics-certs\") pod \"maas-api-79df8fbc7f-x4bsl\" (UID: \"1ed578f8-321b-446d-843c-b022ccc5e5ca\") " pod="odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl"
Sep 10 16:17:07.326920 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.326873    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt9v8\" (UniqueName: \"kubernetes.io/projected/1ed578f8-321b-446d-843c-b022ccc5e5ca-kube-api-access-qt9v8\") pod \"maas-api-79df8fbc7f-x4bsl\" (UID: \"1ed578f8-321b-446d-843c-b022ccc5e5ca\") " pod="odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl"
Sep 10 16:17:07.326920 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.326912    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ed578f8-321b-446d-843c-b022ccc5e5ca-maas-api-tls\") pod \"maas-api-79df8fbc7f-x4bsl\" (UID: \"1ed578f8-321b-446d-843c-b022ccc5e5ca\") " pod="odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl"
Sep 10 16:17:07.428268 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.428227    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ed578f8-321b-446d-843c-b022ccc5e5ca-metrics-certs\") pod \"maas-api-79df8fbc7f-x4bsl\" (UID: \"1ed578f8-321b-446d-843c-b022ccc5e5ca\") " pod="odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl"
Sep 10 16:17:07.428488 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.428285    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt9v8\" (UniqueName: \"kubernetes.io/projected/1ed578f8-321b-446d-843c-b022ccc5e5ca-kube-api-access-qt9v8\") pod \"maas-api-79df8fbc7f-x4bsl\" (UID: \"1ed578f8-321b-446d-843c-b022ccc5e5ca\") " pod="odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl"
Sep 10 16:17:07.428725 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.428691    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ed578f8-321b-446d-843c-b022ccc5e5ca-maas-api-tls\") pod \"maas-api-79df8fbc7f-x4bsl\" (UID: \"1ed578f8-321b-446d-843c-b022ccc5e5ca\") " pod="odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl"
Sep 10 16:17:07.428875 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.428809    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed578f8-321b-446d-843c-b022ccc5e5ca-openshift-service-ca\") pod \"maas-api-79df8fbc7f-x4bsl\" (UID: \"1ed578f8-321b-446d-843c-b022ccc5e5ca\") " pod="odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl"
Sep 10 16:17:07.429626 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.429597    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed578f8-321b-446d-843c-b022ccc5e5ca-openshift-service-ca\") pod \"maas-api-79df8fbc7f-x4bsl\" (UID: \"1ed578f8-321b-446d-843c-b022ccc5e5ca\") " pod="odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl"
Sep 10 16:17:07.431065 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.431043    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ed578f8-321b-446d-843c-b022ccc5e5ca-maas-api-tls\") pod \"maas-api-79df8fbc7f-x4bsl\" (UID: \"1ed578f8-321b-446d-843c-b022ccc5e5ca\") " pod="odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl"
Sep 10 16:17:07.431171 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.431110    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ed578f8-321b-446d-843c-b022ccc5e5ca-metrics-certs\") pod \"maas-api-79df8fbc7f-x4bsl\" (UID: \"1ed578f8-321b-446d-843c-b022ccc5e5ca\") " pod="odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl"
Sep 10 16:17:07.438939 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.438916    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt9v8\" (UniqueName: \"kubernetes.io/projected/1ed578f8-321b-446d-843c-b022ccc5e5ca-kube-api-access-qt9v8\") pod \"maas-api-79df8fbc7f-x4bsl\" (UID: \"1ed578f8-321b-446d-843c-b022ccc5e5ca\") " pod="odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl"
Sep 10 16:17:07.456804 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.456774    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"
Sep 10 16:17:07.530056 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.529955    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf4tc\" (UniqueName: \"kubernetes.io/projected/6948d5ca-0320-44d5-ac2a-17e8968b059b-kube-api-access-lf4tc\") pod \"6948d5ca-0320-44d5-ac2a-17e8968b059b\" (UID: \"6948d5ca-0320-44d5-ac2a-17e8968b059b\") "
Sep 10 16:17:07.530242 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.530084    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6948d5ca-0320-44d5-ac2a-17e8968b059b-openshift-service-ca\") pod \"6948d5ca-0320-44d5-ac2a-17e8968b059b\" (UID: \"6948d5ca-0320-44d5-ac2a-17e8968b059b\") "
Sep 10 16:17:07.530242 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.530112    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6948d5ca-0320-44d5-ac2a-17e8968b059b-metrics-certs\") pod \"6948d5ca-0320-44d5-ac2a-17e8968b059b\" (UID: \"6948d5ca-0320-44d5-ac2a-17e8968b059b\") "
Sep 10 16:17:07.530242 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.530150    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6948d5ca-0320-44d5-ac2a-17e8968b059b-maas-api-tls\") pod \"6948d5ca-0320-44d5-ac2a-17e8968b059b\" (UID: \"6948d5ca-0320-44d5-ac2a-17e8968b059b\") "
Sep 10 16:17:07.530487 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.530462    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6948d5ca-0320-44d5-ac2a-17e8968b059b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6948d5ca-0320-44d5-ac2a-17e8968b059b" (UID: "6948d5ca-0320-44d5-ac2a-17e8968b059b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:17:07.532227 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.532195    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6948d5ca-0320-44d5-ac2a-17e8968b059b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6948d5ca-0320-44d5-ac2a-17e8968b059b" (UID: "6948d5ca-0320-44d5-ac2a-17e8968b059b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:17:07.532365 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.532239    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6948d5ca-0320-44d5-ac2a-17e8968b059b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6948d5ca-0320-44d5-ac2a-17e8968b059b" (UID: "6948d5ca-0320-44d5-ac2a-17e8968b059b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:17:07.532365 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.532247    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6948d5ca-0320-44d5-ac2a-17e8968b059b-kube-api-access-lf4tc" (OuterVolumeSpecName: "kube-api-access-lf4tc") pod "6948d5ca-0320-44d5-ac2a-17e8968b059b" (UID: "6948d5ca-0320-44d5-ac2a-17e8968b059b"). InnerVolumeSpecName "kube-api-access-lf4tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:17:07.613657 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.613613    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl"
Sep 10 16:17:07.632068 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.632027    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf4tc\" (UniqueName: \"kubernetes.io/projected/6948d5ca-0320-44d5-ac2a-17e8968b059b-kube-api-access-lf4tc\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:17:07.632068 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.632070    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6948d5ca-0320-44d5-ac2a-17e8968b059b-openshift-service-ca\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:17:07.632226 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.632086    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6948d5ca-0320-44d5-ac2a-17e8968b059b-metrics-certs\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:17:07.632226 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.632102    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6948d5ca-0320-44d5-ac2a-17e8968b059b-maas-api-tls\") on node \"ip-10-0-134-36.ec2.internal\" DevicePath \"\""
Sep 10 16:17:07.885399 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.885359    2572 generic.go:356] "Generic (PLEG): container finished" podID="6948d5ca-0320-44d5-ac2a-17e8968b059b" containerID="9a27c56d29776c21cefda13d383b1abad55e28c5f4b3b0470b645f553d136590" exitCode=0
Sep 10 16:17:07.885558 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.885418    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"
Sep 10 16:17:07.885558 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.885441    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk" event={"ID":"6948d5ca-0320-44d5-ac2a-17e8968b059b","Type":"ContainerDied","Data":"9a27c56d29776c21cefda13d383b1abad55e28c5f4b3b0470b645f553d136590"}
Sep 10 16:17:07.885558 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.885496    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6689875f55-4x9kk" event={"ID":"6948d5ca-0320-44d5-ac2a-17e8968b059b","Type":"ContainerDied","Data":"c077774f401d9d446654ac58fd751cae3eaafe814055200a6e0765bc2c51d4f5"}
Sep 10 16:17:07.885558 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.885518    2572 scope.go:117] "RemoveContainer" containerID="9a27c56d29776c21cefda13d383b1abad55e28c5f4b3b0470b645f553d136590"
Sep 10 16:17:07.895484 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.895462    2572 scope.go:117] "RemoveContainer" containerID="9a27c56d29776c21cefda13d383b1abad55e28c5f4b3b0470b645f553d136590"
Sep 10 16:17:07.895606 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.895585    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl"]
Sep 10 16:17:07.895796 ip-10-0-134-36 kubenswrapper[2572]: E0910 16:17:07.895779    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a27c56d29776c21cefda13d383b1abad55e28c5f4b3b0470b645f553d136590\": container with ID starting with 9a27c56d29776c21cefda13d383b1abad55e28c5f4b3b0470b645f553d136590 not found: ID does not exist" containerID="9a27c56d29776c21cefda13d383b1abad55e28c5f4b3b0470b645f553d136590"
Sep 10 16:17:07.895841 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.895805    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a27c56d29776c21cefda13d383b1abad55e28c5f4b3b0470b645f553d136590"} err="failed to get container status \"9a27c56d29776c21cefda13d383b1abad55e28c5f4b3b0470b645f553d136590\": rpc error: code = NotFound desc = could not find container \"9a27c56d29776c21cefda13d383b1abad55e28c5f4b3b0470b645f553d136590\": container with ID starting with 9a27c56d29776c21cefda13d383b1abad55e28c5f4b3b0470b645f553d136590 not found: ID does not exist"
Sep 10 16:17:07.898001 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:17:07.897979    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed578f8_321b_446d_843c_b022ccc5e5ca.slice/crio-d0f57bf1e59064921d33525183d619320d85a5afab5b2a322fa0012bc198f870 WatchSource:0}: Error finding container d0f57bf1e59064921d33525183d619320d85a5afab5b2a322fa0012bc198f870: Status 404 returned error can't find the container with id d0f57bf1e59064921d33525183d619320d85a5afab5b2a322fa0012bc198f870
Sep 10 16:17:07.929924 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.929885    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"]
Sep 10 16:17:07.937094 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.937045    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6689875f55-4x9kk"]
Sep 10 16:17:07.994037 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:07.993999    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6948d5ca-0320-44d5-ac2a-17e8968b059b" path="/var/lib/kubelet/pods/6948d5ca-0320-44d5-ac2a-17e8968b059b/volumes"
Sep 10 16:17:08.014199 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:08.014169    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7sm72_8ddd4dc1-1122-41e0-a54a-67fb8008d7b9/ovn-acl-logging/0.log"
Sep 10 16:17:08.024358 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:08.024333    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7sm72_8ddd4dc1-1122-41e0-a54a-67fb8008d7b9/ovn-acl-logging/0.log"
Sep 10 16:17:08.894259 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:08.894215    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl" event={"ID":"1ed578f8-321b-446d-843c-b022ccc5e5ca","Type":"ContainerStarted","Data":"e7d4c2690fdf44be20c78998b528e2b6b2cc1f5f76f94cbd29e1c53efa0960c9"}
Sep 10 16:17:08.894714 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:08.894297    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl" event={"ID":"1ed578f8-321b-446d-843c-b022ccc5e5ca","Type":"ContainerStarted","Data":"d0f57bf1e59064921d33525183d619320d85a5afab5b2a322fa0012bc198f870"}
Sep 10 16:17:08.894714 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:08.894348    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl"
Sep 10 16:17:14.905737 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:14.905707    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl"
Sep 10 16:17:14.943527 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:14.943462    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79df8fbc7f-x4bsl" podStartSLOduration=7.308401658 podStartE2EDuration="7.943445514s" totalImagesPullingTime="635.043856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:08.943840864 +0000 UTC m=+2101.616250959" watchObservedRunningTime="2026-09-10 16:17:14.943445514 +0000 UTC m=+2107.615855596"
Sep 10 16:17:29.593870 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:29.593826    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-565c6b8cd8-p694g_f0099fd3-4549-4e2a-ac0d-9a14da2b3b35/authorino/0.log"
Sep 10 16:17:35.675594 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:35.675560    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-97f7948ff-nthf5_f6a6c38a-156a-4e8b-8870-2fdf8a08a433/manager/0.log"
Sep 10 16:17:35.686799 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:35.686766    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-97f7948ff-nthf5_f6a6c38a-156a-4e8b-8870-2fdf8a08a433/copy-manifests/0.log"
Sep 10 16:17:36.290763 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:36.290727    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79df8fbc7f-x4bsl_1ed578f8-321b-446d-843c-b022ccc5e5ca/maas-api/0.log"
Sep 10 16:17:36.399635 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:36.399599    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817600-j8nrh_9f90761a-4e2a-473a-bc4b-1f2eda9dec27/cleanup/0.log"
Sep 10 16:17:37.383630 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:37.383590    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-565c6b8cd8-p694g_f0099fd3-4549-4e2a-ac0d-9a14da2b3b35/authorino/0.log"
Sep 10 16:17:37.736953 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:37.736869    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-r4gbw_5f160b88-c477-41cf-beed-662a1f108f8d/kuadrant-console-plugin/0.log"
Sep 10 16:17:37.863766 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:37.863729    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-fp8hw_0d8c5820-0773-4326-a315-c44c321b789d/manager/0.log"
Sep 10 16:17:38.726796 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:38.726764    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-644bb569f8-4sghh_8cc11c10-55e2-4a87-ab31-64835557d956/kube-auth-proxy/0.log"
Sep 10 16:17:39.777193 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:39.777143    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-4p8lp_29aa605d-c128-4458-9dc9-a7c0733a4855/main/0.log"
Sep 10 16:17:40.125461 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:40.125433    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzh6x7_62a722d5-5379-4a20-9980-f491822d582c/main/0.log"
Sep 10 16:17:40.359952 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:40.359921    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-4d4rz_ebc04cfb-6032-4bb9-9ec4-5146e22719d8/main/0.log"
Sep 10 16:17:48.965677 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:48.965645    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vz9z7_a7d7c20f-a31c-4468-a822-a7e2601fe3e4/global-pull-secret-syncer/0.log"
Sep 10 16:17:49.123552 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:49.123517    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nf5db_7a6d6658-59b9-48e2-83e1-8a2b8750ecf7/konnectivity-agent/0.log"
Sep 10 16:17:49.176927 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:49.176897    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-36.ec2.internal_872387aaa69ba3ec01d1d25ea3dd5b1d/haproxy/0.log"
Sep 10 16:17:53.582920 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:53.582870    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-565c6b8cd8-p694g_f0099fd3-4549-4e2a-ac0d-9a14da2b3b35/authorino/0.log"
Sep 10 16:17:53.680126 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:53.680093    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-r4gbw_5f160b88-c477-41cf-beed-662a1f108f8d/kuadrant-console-plugin/0.log"
Sep 10 16:17:53.724383 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:53.724344    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-fp8hw_0d8c5820-0773-4326-a315-c44c321b789d/manager/0.log"
Sep 10 16:17:55.162625 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:55.162597    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e14d9389-fb0b-4162-84d9-4a128d7a95c6/alertmanager/0.log"
Sep 10 16:17:55.187234 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:55.187203    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e14d9389-fb0b-4162-84d9-4a128d7a95c6/config-reloader/0.log"
Sep 10 16:17:55.210104 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:55.210075    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e14d9389-fb0b-4162-84d9-4a128d7a95c6/kube-rbac-proxy-web/0.log"
Sep 10 16:17:55.247172 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:55.247139    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e14d9389-fb0b-4162-84d9-4a128d7a95c6/kube-rbac-proxy/0.log"
Sep 10 16:17:55.275538 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:55.275512    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e14d9389-fb0b-4162-84d9-4a128d7a95c6/kube-rbac-proxy-metric/0.log"
Sep 10 16:17:55.306400 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:55.306373    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e14d9389-fb0b-4162-84d9-4a128d7a95c6/prom-label-proxy/0.log"
Sep 10 16:17:55.342833 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:55.342803    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e14d9389-fb0b-4162-84d9-4a128d7a95c6/init-config-reloader/0.log"
Sep 10 16:17:55.529810 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:55.529727    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-8lm2h_37b85e59-be69-499c-85f4-80822b57748b/monitoring-plugin/0.log"
Sep 10 16:17:55.671840 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:55.671810    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-96zqr_ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f/node-exporter/0.log"
Sep 10 16:17:55.695020 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:55.694987    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-96zqr_ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f/kube-rbac-proxy/0.log"
Sep 10 16:17:55.724860 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:55.724827    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-96zqr_ab6b3fa4-7917-4dd5-bc85-7b6bdbe3802f/init-textfile/0.log"
Sep 10 16:17:56.183506 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:56.183422    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-xv7bn_6fc7de00-4d51-42c5-b8dd-7077840e7d79/prometheus-operator/0.log"
Sep 10 16:17:56.209094 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:56.209032    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-xv7bn_6fc7de00-4d51-42c5-b8dd-7077840e7d79/kube-rbac-proxy/0.log"
Sep 10 16:17:56.383858 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:56.383827    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b74c4f759-qjbvs_040f0003-1267-49e6-a365-70608c4bc7d1/thanos-query/0.log"
Sep 10 16:17:56.412124 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:56.412096    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b74c4f759-qjbvs_040f0003-1267-49e6-a365-70608c4bc7d1/kube-rbac-proxy-web/0.log"
Sep 10 16:17:56.439870 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:56.439794    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b74c4f759-qjbvs_040f0003-1267-49e6-a365-70608c4bc7d1/kube-rbac-proxy/0.log"
Sep 10 16:17:56.476928 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:56.476868    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b74c4f759-qjbvs_040f0003-1267-49e6-a365-70608c4bc7d1/prom-label-proxy/0.log"
Sep 10 16:17:56.503252 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:56.503223    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b74c4f759-qjbvs_040f0003-1267-49e6-a365-70608c4bc7d1/kube-rbac-proxy-rules/0.log"
Sep 10 16:17:56.529244 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:56.529218    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b74c4f759-qjbvs_040f0003-1267-49e6-a365-70608c4bc7d1/kube-rbac-proxy-metrics/0.log"
Sep 10 16:17:59.187268 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.187232    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"]
Sep 10 16:17:59.187686 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.187598    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6948d5ca-0320-44d5-ac2a-17e8968b059b" containerName="maas-api"
Sep 10 16:17:59.187686 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.187611    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6948d5ca-0320-44d5-ac2a-17e8968b059b" containerName="maas-api"
Sep 10 16:17:59.187686 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.187675    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6948d5ca-0320-44d5-ac2a-17e8968b059b" containerName="maas-api"
Sep 10 16:17:59.190694 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.190671    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"
Sep 10 16:17:59.193523 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.193500    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t86vd\"/\"kube-root-ca.crt\""
Sep 10 16:17:59.194097 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.194076    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t86vd\"/\"default-dockercfg-b47qk\""
Sep 10 16:17:59.194176 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.194118    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t86vd\"/\"openshift-service-ca.crt\""
Sep 10 16:17:59.205595 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.205563    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"]
Sep 10 16:17:59.281276 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.281230    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dcd9c06f-c587-4c27-a64c-b488ca3f7d21-podres\") pod \"perf-node-gather-daemonset-nmwb2\" (UID: \"dcd9c06f-c587-4c27-a64c-b488ca3f7d21\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"
Sep 10 16:17:59.281481 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.281285    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n8np\" (UniqueName: \"kubernetes.io/projected/dcd9c06f-c587-4c27-a64c-b488ca3f7d21-kube-api-access-6n8np\") pod \"perf-node-gather-daemonset-nmwb2\" (UID: \"dcd9c06f-c587-4c27-a64c-b488ca3f7d21\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"
Sep 10 16:17:59.281481 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.281386    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dcd9c06f-c587-4c27-a64c-b488ca3f7d21-proc\") pod \"perf-node-gather-daemonset-nmwb2\" (UID: \"dcd9c06f-c587-4c27-a64c-b488ca3f7d21\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"
Sep 10 16:17:59.281481 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.281450    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dcd9c06f-c587-4c27-a64c-b488ca3f7d21-lib-modules\") pod \"perf-node-gather-daemonset-nmwb2\" (UID: \"dcd9c06f-c587-4c27-a64c-b488ca3f7d21\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"
Sep 10 16:17:59.281481 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.281477    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dcd9c06f-c587-4c27-a64c-b488ca3f7d21-sys\") pod \"perf-node-gather-daemonset-nmwb2\" (UID: \"dcd9c06f-c587-4c27-a64c-b488ca3f7d21\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"
Sep 10 16:17:59.382795 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.382747    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dcd9c06f-c587-4c27-a64c-b488ca3f7d21-proc\") pod \"perf-node-gather-daemonset-nmwb2\" (UID: \"dcd9c06f-c587-4c27-a64c-b488ca3f7d21\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"
Sep 10 16:17:59.383012 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.382820    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dcd9c06f-c587-4c27-a64c-b488ca3f7d21-lib-modules\") pod \"perf-node-gather-daemonset-nmwb2\" (UID: \"dcd9c06f-c587-4c27-a64c-b488ca3f7d21\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"
Sep 10 16:17:59.383012 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.382853    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dcd9c06f-c587-4c27-a64c-b488ca3f7d21-sys\") pod \"perf-node-gather-daemonset-nmwb2\" (UID: \"dcd9c06f-c587-4c27-a64c-b488ca3f7d21\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"
Sep 10 16:17:59.383012 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.382890    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dcd9c06f-c587-4c27-a64c-b488ca3f7d21-podres\") pod \"perf-node-gather-daemonset-nmwb2\" (UID: \"dcd9c06f-c587-4c27-a64c-b488ca3f7d21\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"
Sep 10 16:17:59.383012 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.382887    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dcd9c06f-c587-4c27-a64c-b488ca3f7d21-proc\") pod \"perf-node-gather-daemonset-nmwb2\" (UID: \"dcd9c06f-c587-4c27-a64c-b488ca3f7d21\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"
Sep 10 16:17:59.383012 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.382912    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n8np\" (UniqueName: \"kubernetes.io/projected/dcd9c06f-c587-4c27-a64c-b488ca3f7d21-kube-api-access-6n8np\") pod \"perf-node-gather-daemonset-nmwb2\" (UID: \"dcd9c06f-c587-4c27-a64c-b488ca3f7d21\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"
Sep 10 16:17:59.383012 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.382939    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dcd9c06f-c587-4c27-a64c-b488ca3f7d21-lib-modules\") pod \"perf-node-gather-daemonset-nmwb2\" (UID: \"dcd9c06f-c587-4c27-a64c-b488ca3f7d21\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"
Sep 10 16:17:59.383012 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.382972    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dcd9c06f-c587-4c27-a64c-b488ca3f7d21-sys\") pod \"perf-node-gather-daemonset-nmwb2\" (UID: \"dcd9c06f-c587-4c27-a64c-b488ca3f7d21\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"
Sep 10 16:17:59.383012 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.383007    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dcd9c06f-c587-4c27-a64c-b488ca3f7d21-podres\") pod \"perf-node-gather-daemonset-nmwb2\" (UID: \"dcd9c06f-c587-4c27-a64c-b488ca3f7d21\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"
Sep 10 16:17:59.392443 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.392416    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n8np\" (UniqueName: \"kubernetes.io/projected/dcd9c06f-c587-4c27-a64c-b488ca3f7d21-kube-api-access-6n8np\") pod \"perf-node-gather-daemonset-nmwb2\" (UID: \"dcd9c06f-c587-4c27-a64c-b488ca3f7d21\") " pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"
Sep 10 16:17:59.501521 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.501423    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"
Sep 10 16:17:59.784928 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:17:59.784688    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"]
Sep 10 16:17:59.787790 ip-10-0-134-36 kubenswrapper[2572]: W0910 16:17:59.787761    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddcd9c06f_c587_4c27_a64c_b488ca3f7d21.slice/crio-fcf0b1db8eff86e6c3d30e0ec4c5e00f4ce58cbf49f137497db0966ab2c9bd0b WatchSource:0}: Error finding container fcf0b1db8eff86e6c3d30e0ec4c5e00f4ce58cbf49f137497db0966ab2c9bd0b: Status 404 returned error can't find the container with id fcf0b1db8eff86e6c3d30e0ec4c5e00f4ce58cbf49f137497db0966ab2c9bd0b
Sep 10 16:18:00.172619 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:00.172577    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2" event={"ID":"dcd9c06f-c587-4c27-a64c-b488ca3f7d21","Type":"ContainerStarted","Data":"f8e8ef1a4d1d97fb6915dcf170ce570006dec10e3ab29cf864ec5f04efc98277"}
Sep 10 16:18:00.172819 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:00.172649    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2" event={"ID":"dcd9c06f-c587-4c27-a64c-b488ca3f7d21","Type":"ContainerStarted","Data":"fcf0b1db8eff86e6c3d30e0ec4c5e00f4ce58cbf49f137497db0966ab2c9bd0b"}
Sep 10 16:18:00.172819 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:00.172675    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"
Sep 10 16:18:00.193421 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:00.193366    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2" podStartSLOduration=1.193346555 podStartE2EDuration="1.193346555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:00.191103438 +0000 UTC m=+2152.863513518" watchObservedRunningTime="2026-09-10 16:18:00.193346555 +0000 UTC m=+2152.865756639"
Sep 10 16:18:01.388442 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:01.388394    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xrtrf_97b545a8-3fac-4bcf-b67c-f51a919e4508/dns/0.log"
Sep 10 16:18:01.430913 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:01.430885    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xrtrf_97b545a8-3fac-4bcf-b67c-f51a919e4508/kube-rbac-proxy/0.log"
Sep 10 16:18:01.555125 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:01.555092    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-54n4f_1611fac1-d845-4d08-bdbf-72957655ab60/dns-node-resolver/0.log"
Sep 10 16:18:02.129557 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:02.129527    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d854b8bbc-pv7h9_721c0590-d431-4abf-b9b6-fd5ab568ddcd/registry/0.log"
Sep 10 16:18:02.160491 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:02.160457    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q944h_f221b59a-f955-4ce1-a9b4-e4efe942a24a/node-ca/0.log"
Sep 10 16:18:03.701600 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:03.701553    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-644bb569f8-4sghh_8cc11c10-55e2-4a87-ab31-64835557d956/kube-auth-proxy/0.log"
Sep 10 16:18:04.986152 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:04.986120    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nkbvh_6b4a2209-e010-48a8-b869-89467731e086/serve-healthcheck-canary/0.log"
Sep 10 16:18:05.745405 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:05.745371    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vclm6_06cdbeec-2930-46fd-bf24-51c86356c801/kube-rbac-proxy/0.log"
Sep 10 16:18:05.791363 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:05.791333    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vclm6_06cdbeec-2930-46fd-bf24-51c86356c801/exporter/0.log"
Sep 10 16:18:05.839189 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:05.839156    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vclm6_06cdbeec-2930-46fd-bf24-51c86356c801/extractor/0.log"
Sep 10 16:18:06.187716 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:06.187685    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t86vd/perf-node-gather-daemonset-nmwb2"
Sep 10 16:18:08.578193 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:08.578151    2572 scope.go:117] "RemoveContainer" containerID="b897ec3ba02027c90b48c6abafa459dab00f18c8a16bbc861f04fdd0a3b4fd26"
Sep 10 16:18:09.178432 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:09.178406    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-97f7948ff-nthf5_f6a6c38a-156a-4e8b-8870-2fdf8a08a433/manager/0.log"
Sep 10 16:18:09.204846 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:09.204814    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-97f7948ff-nthf5_f6a6c38a-156a-4e8b-8870-2fdf8a08a433/copy-manifests/0.log"
Sep 10 16:18:09.998012 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:09.997982    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-644bb569f8-4sghh_8cc11c10-55e2-4a87-ab31-64835557d956/kube-auth-proxy/0.log"
Sep 10 16:18:10.655095 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:10.655064    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-79dbb968b5-45wr5_787c6c8d-daa0-4bb3-967a-98fb1cc42d6d/manager/0.log"
Sep 10 16:18:12.183976 ip-10-0-134-36 kubenswrapper[2572]: I0910 16:18:12.183945    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-644bb569f8-4sghh_8cc11c10-55e2-4a87-ab31-64835557d956/kube-auth-proxy/0.log"