Sep 19 11:33:34.396263 ip-10-0-133-209 systemd[1]: Starting Kubernetes Kubelet...
Sep 19 11:33:37.727440 ip-10-0-133-209 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 11:33:37.727440 ip-10-0-133-209 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 19 11:33:37.727440 ip-10-0-133-209 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 11:33:37.727440 ip-10-0-133-209 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 19 11:33:37.727440 ip-10-0-133-209 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 11:33:37.728323 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.727519    2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 19 11:33:37.731258 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731231    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 11:33:37.731258 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731255    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 11:33:37.731258 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731260    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 11:33:37.731258 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731264    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 11:33:37.731258 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731268    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731272    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731276    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731279    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731283    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731286    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731290    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731293    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731305    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731310    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731313    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731317    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731320    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731323    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731327    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731331    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731335    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731339    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731343    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731346    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 11:33:37.731557 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731350    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 11:33:37.732372 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731353    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 11:33:37.732372 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731357    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 11:33:37.732372 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731360    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 11:33:37.732372 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731364    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 11:33:37.732372 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731368    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 11:33:37.732372 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731371    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 11:33:37.732372 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731375    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 11:33:37.732372 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731379    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 11:33:37.732372 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731382    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 11:33:37.732372 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731386    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 11:33:37.732372 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731390    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 11:33:37.732372 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731393    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 11:33:37.732372 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731397    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 11:33:37.732372 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731402    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 11:33:37.732372 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731406    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 11:33:37.732372 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731410    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 11:33:37.732372 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731414    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 11:33:37.732372 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731419    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 11:33:37.732372 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731424    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731428    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731433    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731437    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731441    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731444    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731448    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731452    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731456    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731460    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731464    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731468    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731472    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731476    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731480    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731484    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731488    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731492    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731496    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731500    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 11:33:37.733203 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731504    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731508    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731513    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731517    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731522    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731528    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731532    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731537    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731541    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731545    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731549    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731553    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731559    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731566    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731572    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731577    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731581    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731586    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731590    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731595    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 11:33:37.734008 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731599    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 11:33:37.734877 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731604    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 11:33:37.734877 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731608    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 11:33:37.734877 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731612    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 11:33:37.734877 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731616    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 11:33:37.734877 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731619    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 11:33:37.734877 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731624    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 11:33:37.734877 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731630    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 11:33:37.734877 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731638    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 11:33:37.734877 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731645    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 11:33:37.734877 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731650    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 11:33:37.734877 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731654    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 11:33:37.734877 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731659    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 11:33:37.734877 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731663    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 11:33:37.734877 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731668    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 11:33:37.734877 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.731673    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 11:33:37.734877 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.732900    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 11:33:37.734877 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.732934    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 11:33:37.734877 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.732941    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.732947    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.732952    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.732957    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.732962    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.732966    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.732970    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.732975    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.732980    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.732984    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.732989    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.732993    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.732997    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733000    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733005    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733009    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733013    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733017    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733021    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733025    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 11:33:37.735632 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733029    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 11:33:37.736427 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733033    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 11:33:37.736427 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733038    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 11:33:37.736427 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733042    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 11:33:37.736427 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733046    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 11:33:37.736427 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733050    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 11:33:37.736427 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733057    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 11:33:37.736427 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733062    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 11:33:37.736427 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733066    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 11:33:37.736427 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733071    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 11:33:37.736427 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733076    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 11:33:37.736427 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733080    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 11:33:37.736427 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733085    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 11:33:37.736427 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733090    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 11:33:37.736427 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733095    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 11:33:37.736427 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733099    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 11:33:37.736427 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733103    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 11:33:37.736427 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733107    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 11:33:37.736427 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733111    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 11:33:37.736427 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733115    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 11:33:37.736936 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733120    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 11:33:37.736936 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733124    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 11:33:37.736936 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733129    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 11:33:37.736936 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733133    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 11:33:37.736936 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733137    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 11:33:37.736936 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733141    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 11:33:37.736936 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733145    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 11:33:37.736936 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733149    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 11:33:37.736936 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733153    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 11:33:37.736936 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733157    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 11:33:37.736936 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733161    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 11:33:37.736936 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733165    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 11:33:37.736936 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733171    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 11:33:37.736936 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733175    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 11:33:37.736936 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733179    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 11:33:37.736936 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733185    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 11:33:37.736936 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733191    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 11:33:37.736936 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733196    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 11:33:37.736936 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733202    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733205    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733209    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733213    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733217    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733221    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733225    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733230    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733234    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733238    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733242    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733246    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733251    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733255    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733259    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733263    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733268    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733271    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733276    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733280    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 11:33:37.737415 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733283    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733287    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733291    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733295    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733299    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733303    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733307    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733311    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733315    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733318    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733322    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733326    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733331    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733335    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733339    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733343    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733347    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733350    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.733354    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734001    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 11:33:37.738028 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734012    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734017    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734022    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734027    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734031    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734036    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734040    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734046    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734050    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734055    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734059    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734063    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734067    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734071    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734075    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734079    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734083    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734087    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734091    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734095    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 11:33:37.738634 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734099    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734103    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734107    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734111    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734116    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734119    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734123    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734127    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734132    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734136    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734140    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734147    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734152    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734156    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734160    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734163    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734168    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734172    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734176    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734180    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 11:33:37.739130 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734185    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 11:33:37.739690 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734191    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 11:33:37.739690 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734195    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 11:33:37.739690 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734199    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 11:33:37.739690 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734203    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 11:33:37.739690 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734207    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 11:33:37.739690 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734211    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 11:33:37.739690 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734215    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 11:33:37.739690 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734219    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 11:33:37.739690 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734223    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 11:33:37.739690 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734228    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 11:33:37.739690 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734232    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 11:33:37.739690 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734236    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 11:33:37.739690 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734240    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 11:33:37.739690 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734244    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 11:33:37.739690 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734248    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 11:33:37.739690 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734252    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 11:33:37.739690 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734256    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 11:33:37.739690 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734261    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 11:33:37.739690 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734265    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734269    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734273    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734278    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734285    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734288    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734292    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734296    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734301    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734304    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734308    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734312    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734316    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734320    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734325    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734330    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734334    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734338    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734342    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734346    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 11:33:37.740167 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734350    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734354    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734358    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734365    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734370    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734374    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734379    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734383    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734388    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734393    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734397    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734401    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734405    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734409    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734413    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734417    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734423    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734427    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.734431    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734559    2584 flags.go:64] FLAG: --address="0.0.0.0"
Sep 19 11:33:37.740638 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734618    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734629    2584 flags.go:64] FLAG: --anonymous-auth="true"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734636    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734642    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734648    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734655    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734661    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734668    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734673    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734678    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734683    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734689    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734694    2584 flags.go:64] FLAG: --cgroup-root=""
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734698    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734703    2584 flags.go:64] FLAG: --client-ca-file=""
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734708    2584 flags.go:64] FLAG: --cloud-provider="external"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734712    2584 flags.go:64] FLAG: --cluster-dns="[]"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734718    2584 flags.go:64] FLAG: --cluster-domain=""
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734723    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734728    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734733    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734739    2584 flags.go:64] FLAG: --container-log-max-files="5"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734744    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 19 11:33:37.741138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734749    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734754    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734760    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734765    2584 flags.go:64] FLAG: --contention-profiling="false"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734769    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734777    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734782    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734787    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734794    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734798    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734803    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734807    2584 flags.go:64] FLAG: --enable-load-reader="false"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734813    2584 flags.go:64] FLAG: --enable-server="true"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734817    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734825    2584 flags.go:64] FLAG: --event-burst="100"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734830    2584 flags.go:64] FLAG: --event-qps="50"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734834    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734840    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734846    2584 flags.go:64] FLAG: --eviction-hard=""
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734852    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734856    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734861    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734866    2584 flags.go:64] FLAG: --eviction-soft=""
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734871    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734875    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 19 11:33:37.741685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734879    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734883    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734888    2584 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734893    2584 flags.go:64] FLAG: --fail-swap-on="true"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734899    2584 flags.go:64] FLAG: --feature-gates=""
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734905    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734928    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734934    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734939    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734944    2584 flags.go:64] FLAG: --healthz-port="10248"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734949    2584 flags.go:64] FLAG: --help="false"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734954    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-133-209.ec2.internal"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734959    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734963    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734967    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734973    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734978    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734982    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734987    2584 flags.go:64] FLAG: --image-service-endpoint=""
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734992    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.734996    2584 flags.go:64] FLAG: --kube-api-burst="100"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735001    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735005    2584 flags.go:64] FLAG: --kube-api-qps="50"
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735010    2584 flags.go:64] FLAG: --kube-reserved=""
Sep 19 11:33:37.742390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735015    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735021    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735025    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735030    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735034    2584 flags.go:64] FLAG: --lock-file=""
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735039    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735044    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735048    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735057    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735062    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735066    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735071    2584 flags.go:64] FLAG: --logging-format="text"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735077    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735083    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735087    2584 flags.go:64] FLAG: --manifest-url=""
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735092    2584 flags.go:64] FLAG: --manifest-url-header=""
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735099    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735104    2584 flags.go:64] FLAG: --max-open-files="1000000"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735111    2584 flags.go:64] FLAG: --max-pods="110"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735115    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735121    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735125    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735130    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735135    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735140    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 19 11:33:37.743068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735144    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735157    2584 flags.go:64] FLAG: --node-status-max-images="50"
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735162    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735166    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735171    2584 flags.go:64] FLAG: --pod-cidr=""
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735176    2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735185    2584 flags.go:64] FLAG: --pod-manifest-path=""
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735190    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735195    2584 flags.go:64] FLAG: --pods-per-core="0"
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735200    2584 flags.go:64] FLAG: --port="10250"
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735205    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735209    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05aa04fbef2dd8d1d"
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735214    2584 flags.go:64] FLAG: --qos-reserved=""
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735219    2584 flags.go:64] FLAG: --read-only-port="10255"
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735224    2584 flags.go:64] FLAG: --register-node="true"
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735228    2584 flags.go:64] FLAG: --register-with-taints=""
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735234    2584 flags.go:64] FLAG: --registry-burst="10"
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735239    2584 flags.go:64] FLAG: --registry-qps="5"
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735243    2584 flags.go:64] FLAG: --reserved-cpus=""
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735249    2584 flags.go:64] FLAG: --reserved-memory=""
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735255    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735260    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735266    2584 flags.go:64] FLAG: --rotate-certificates="false"
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735270    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735274    2584 flags.go:64] FLAG: --runonce="false"
Sep 19 11:33:37.743706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735279    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735284    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735289    2584 flags.go:64] FLAG: --seccomp-default="false"
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735294    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735298    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735303    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735308    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735313    2584 flags.go:64] FLAG: --storage-driver-password="root"
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735331    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735337    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735341    2584 flags.go:64] FLAG: --storage-driver-user="root"
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735346    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735351    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735356    2584 flags.go:64] FLAG: --system-cgroups=""
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735360    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735369    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735373    2584 flags.go:64] FLAG: --tls-cert-file=""
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735379    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735385    2584 flags.go:64] FLAG: --tls-min-version=""
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735389    2584 flags.go:64] FLAG: --tls-private-key-file=""
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735393    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735398    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735402    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735407    2584 flags.go:64] FLAG: --v="2"
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735413    2584 flags.go:64] FLAG: --version="false"
Sep 19 11:33:37.744336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735419    2584 flags.go:64] FLAG: --vmodule=""
Sep 19 11:33:37.744971 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735426    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 19 11:33:37.744971 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.735431    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 19 11:33:37.744971 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735624    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 11:33:37.744971 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735631    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 11:33:37.744971 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735635    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 11:33:37.744971 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735641    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 11:33:37.744971 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735647    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 11:33:37.744971 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735653    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 11:33:37.744971 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735658    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 11:33:37.744971 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735663    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 11:33:37.744971 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735668    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 11:33:37.744971 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735673    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 11:33:37.744971 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735677    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 11:33:37.744971 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735681    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 11:33:37.744971 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735686    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 11:33:37.744971 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735690    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 11:33:37.744971 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735694    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 11:33:37.744971 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735698    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 11:33:37.744971 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735703    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735707    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735711    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735715    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735719    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735723    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735729    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735733    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735737    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735741    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735745    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735749    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735752    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735756    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735762    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735766    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735770    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735774    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735778    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735782    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 11:33:37.745437 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735786    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 11:33:37.745973 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735791    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 11:33:37.745973 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735797    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 11:33:37.745973 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735801    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 11:33:37.745973 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735805    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 11:33:37.745973 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735811    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 11:33:37.745973 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735816    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 11:33:37.745973 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735820    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 11:33:37.745973 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735825    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 11:33:37.745973 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735829    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 11:33:37.745973 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735833    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 11:33:37.745973 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735837    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 11:33:37.745973 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735841    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 11:33:37.745973 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735845    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 11:33:37.745973 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735849    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 11:33:37.745973 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735853    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 11:33:37.745973 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735857    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 11:33:37.745973 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735861    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 11:33:37.745973 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735865    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 11:33:37.745973 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735871    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 11:33:37.746436 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735875    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 11:33:37.746436 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735879    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 11:33:37.746436 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735885    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 11:33:37.746436 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735890    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 11:33:37.746436 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735895    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 11:33:37.746436 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735899    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 11:33:37.746436 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735906    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 11:33:37.746436 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735929    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 11:33:37.746436 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735933    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 11:33:37.746436 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735937    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 11:33:37.746436 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735941    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 11:33:37.746436 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735946    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 11:33:37.746436 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735950    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 11:33:37.746436 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735955    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 11:33:37.746436 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735961    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 11:33:37.746436 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735964    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 11:33:37.746436 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735969    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 11:33:37.746436 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735973    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 11:33:37.746436 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735977    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735981    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735985    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735989    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735993    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.735997    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.736001    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.736005    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.736008    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.736012    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.736016    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.736021    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.736025    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.736029    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.736039    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.736043    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.736047    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.736052    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.736056    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.736060    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 11:33:37.746905 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.736066    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 11:33:37.747703 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.736069    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 11:33:37.747703 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.736074    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 11:33:37.747703 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.736078    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 11:33:37.747703 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.736082    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 11:33:37.747703 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.736090    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 11:33:37.747703 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.744381    2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 19 11:33:37.747703 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.744402    2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 19 11:33:37.747703 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.744423    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 19 11:33:37.747703 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.744432    2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 19 11:33:37.747703 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744495    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 11:33:37.747703 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744501    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 11:33:37.747703 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744505    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 11:33:37.747703 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744507    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 11:33:37.747703 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744510    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 11:33:37.748155 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744513    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 11:33:37.748155 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744516    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 11:33:37.748155 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744519    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 11:33:37.748155 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744522    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 11:33:37.748155 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744524    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 11:33:37.748155 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744527    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 11:33:37.748155 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744529    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 11:33:37.748155 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744532    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 11:33:37.748155 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744535    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 11:33:37.748155 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744537    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 11:33:37.748155 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744540    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 11:33:37.748155 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744543    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 11:33:37.748155 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744546    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 11:33:37.748155 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744548    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 11:33:37.748155 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744550    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 11:33:37.748155 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744553    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 11:33:37.748155 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744555    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 11:33:37.748155 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744558    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 11:33:37.748155 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744560    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744562    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744565    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744567    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744569    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744572    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744582    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744585    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744587    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744590    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744592    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744595    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744598    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744600    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744603    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744605    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744608    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744610    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744614    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744616    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 11:33:37.748660 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744618    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744621    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744623    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744626    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744628    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744631    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744633    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744635    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744639    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744641    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744644    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744646    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744648    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744651    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744653    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744656    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744658    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744661    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744664    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744674    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 11:33:37.749301 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744678    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744682    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744684    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744687    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744691    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744694    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744696    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744699    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744702    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744704    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744707    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744709    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744712    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744714    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744716    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744719    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744721    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744724    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744726    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744729    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 11:33:37.749836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744731    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 11:33:37.750382 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744734    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 11:33:37.750382 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744736    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 11:33:37.750382 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744739    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 11:33:37.750382 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744741    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 11:33:37.750382 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744749    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 11:33:37.750382 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744752    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 11:33:37.750382 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744755    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 11:33:37.750382 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744757    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 11:33:37.750382 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744759    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 11:33:37.750382 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744762    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 11:33:37.750382 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744765    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 11:33:37.750382 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744773    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 11:33:37.750382 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744775    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 11:33:37.750382 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744779    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 11:33:37.750382 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744783    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 11:33:37.750748 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.744788    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 11:33:37.750748 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744947    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 11:33:37.750748 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744954    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 11:33:37.750748 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744957    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 11:33:37.750748 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744960    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 11:33:37.750748 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744962    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 11:33:37.750748 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744965    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 11:33:37.750748 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744968    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 11:33:37.750748 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744970    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 11:33:37.750748 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744973    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 11:33:37.750748 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744975    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 11:33:37.750748 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744978    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 11:33:37.750748 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744981    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 11:33:37.750748 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744984    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 11:33:37.750748 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744986    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744989    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744992    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744994    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744997    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.744999    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745001    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745004    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745006    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745009    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745011    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745014    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745016    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745019    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745021    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745030    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745032    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745035    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745037    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745039    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 11:33:37.751254 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745042    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 11:33:37.751723 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745044    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 11:33:37.751723 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745047    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 11:33:37.751723 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745049    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 11:33:37.751723 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745052    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 11:33:37.751723 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745054    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 11:33:37.751723 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745057    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 11:33:37.751723 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745060    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 11:33:37.751723 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745062    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 11:33:37.751723 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745065    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 11:33:37.751723 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745068    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 11:33:37.751723 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745071    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 11:33:37.751723 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745075    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 11:33:37.751723 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745077    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 11:33:37.751723 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745080    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 11:33:37.751723 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745082    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 11:33:37.751723 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745084    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 11:33:37.751723 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745087    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 11:33:37.751723 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745090    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 11:33:37.751723 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745094    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745097    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745099    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745101    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745104    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745107    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745109    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745112    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745114    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745122    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745125    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745127    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745130    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745132    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745134    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745137    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745140    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745142    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745145    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745148    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 11:33:37.752193 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745150    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 11:33:37.752680 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745153    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 11:33:37.752680 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745155    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 11:33:37.752680 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745158    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 11:33:37.752680 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745160    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 11:33:37.752680 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745162    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 11:33:37.752680 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745165    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 11:33:37.752680 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745167    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 11:33:37.752680 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745170    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 11:33:37.752680 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745173    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 11:33:37.752680 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745177    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 11:33:37.752680 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745179    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 11:33:37.752680 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745182    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 11:33:37.752680 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745185    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 11:33:37.752680 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745187    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 11:33:37.752680 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745190    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 11:33:37.752680 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745192    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 11:33:37.752680 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745195    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 11:33:37.752680 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745198    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 11:33:37.752680 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745200    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 11:33:37.753158 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745202    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 11:33:37.753158 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745205    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 11:33:37.753158 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745213    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 11:33:37.753158 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745216    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 11:33:37.753158 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745218    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 11:33:37.753158 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745220    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 11:33:37.753158 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:37.745223    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 11:33:37.753158 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.745228    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 11:33:37.753158 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.745383    2584 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 19 11:33:37.753158 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.749827    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 19 11:33:37.753158 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.750989    2584 server.go:1019] "Starting client certificate rotation"
Sep 19 11:33:37.753158 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.751087    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 11:33:37.753158 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.751133    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 11:33:37.753516 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.753358    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 11:33:37.755380 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.755345    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 11:33:37.765332 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.765309    2584 log.go:25] "Validated CRI v1 runtime API"
Sep 19 11:33:37.768066 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.768044    2584 log.go:25] "Validated CRI v1 image API"
Sep 19 11:33:37.768777 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.768760    2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 19 11:33:37.776289 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.776250    2584 fs.go:135] Filesystem UUIDs: map[6b8c3272-0745-4d53-9742-006b8dc8b266:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ad9411b8-4e4f-409d-891e-6d30ed53ca55:/dev/nvme0n1p4]
Sep 19 11:33:37.776289 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.776284    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 19 11:33:37.782355 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.782212    2584 manager.go:216] Machine: {Timestamp:2026-09-19 11:33:37.781701418 +0000 UTC m=+1.981731213 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2985610 MemoryCapacity:32812146688 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec290cae763f1956a75ce6493fec2734 SystemUUID:ec290cae-763f-1956-a75c-e6493fec2734 BootID:b68faa12-9adc-4414-8e66-5638b63bc175 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406073344 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c9:1a:6c:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c9:1a:6c:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:0d:95:04:57:7f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812146688 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 19 11:33:37.782355 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.782335    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 19 11:33:37.782536 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.782450    2584 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 19 11:33:37.784119 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.784088    2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 19 11:33:37.784267 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.784118    2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-209.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 19 11:33:37.785027 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.785010    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 11:33:37.785317 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.785306    2584 topology_manager.go:138] "Creating topology manager with none policy"
Sep 19 11:33:37.785368 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.785319    2584 container_manager_linux.go:309] "Creating device plugin manager"
Sep 19 11:33:37.785368 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.785327    2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 11:33:37.785368 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.785337    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 11:33:37.786666 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.786656    2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 19 11:33:37.790501 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.790487    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 11:33:37.790623 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.790614    2584 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 19 11:33:37.790651 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.790645    2584 kubelet.go:486] "Attempting to sync node with API server"
Sep 19 11:33:37.790687 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.790654    2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 19 11:33:37.790687 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.790667    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 19 11:33:37.790687 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.790677    2584 kubelet.go:398] "Adding apiserver pod source"
Sep 19 11:33:37.790687 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.790687    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 19 11:33:37.791080 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.791068    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 11:33:37.791123 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.791086    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 11:33:37.794437 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.794413    2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 19 11:33:37.796824 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.796807    2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 19 11:33:37.796892 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.796837    2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 19 11:33:37.797023 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.797013    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 19 11:33:37.797061 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.797028    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 19 11:33:37.797061 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.797034    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 19 11:33:37.797061 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.797039    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 19 11:33:37.797061 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.797047    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 19 11:33:37.797061 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.797052    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 19 11:33:37.797637 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.797623    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 19 11:33:37.797637 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.797636    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 19 11:33:37.797704 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.797644    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 19 11:33:37.797704 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.797649    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 19 11:33:37.797704 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.797659    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 19 11:33:37.797704 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.797668    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 19 11:33:37.797704 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.797694    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 19 11:33:37.797704 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.797703    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 19 11:33:37.801413 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.801394    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-209.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 19 11:33:37.802248 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.802233    2584 server.go:1268] "Started kubelet"
Sep 19 11:33:37.802329 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:37.802309    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 19 11:33:37.802383 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:37.802309    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-209.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 19 11:33:37.802442 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.802336    2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 19 11:33:37.802442 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.802368    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 19 11:33:37.802516 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.802443    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 19 11:33:37.802778 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.802759    2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 19 11:33:37.803187 ip-10-0-133-209 systemd[1]: Started Kubernetes Kubelet.
Sep 19 11:33:37.804835 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.804817    2584 server.go:310] "Adding debug handlers to kubelet server"
Sep 19 11:33:37.806365 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:37.806346    2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 19 11:33:37.811839 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.811821    2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 19 11:33:37.811936 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.811869    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 19 11:33:37.812114 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:37.812063    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-209.ec2.internal\" not found"
Sep 19 11:33:37.812471 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.812455    2584 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 19 11:33:37.812471 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.812474    2584 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 19 11:33:37.812603 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.812461    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 19 11:33:37.812603 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.812571    2584 reconstruct.go:97] "Volume reconstruction finished"
Sep 19 11:33:37.812603 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.812579    2584 reconciler.go:29] "Reconciler: start to sync state"
Sep 19 11:33:37.812730 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.812717    2584 factory.go:55] Registering systemd factory
Sep 19 11:33:37.812730 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.812732    2584 factory.go:223] Registration of the systemd container factory successfully
Sep 19 11:33:37.812976 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.812961    2584 factory.go:153] Registering CRI-O factory
Sep 19 11:33:37.812976 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.812980    2584 factory.go:223] Registration of the crio container factory successfully
Sep 19 11:33:37.813105 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.813035    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 19 11:33:37.813105 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.813049    2584 factory.go:103] Registering Raw factory
Sep 19 11:33:37.813105 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.813064    2584 manager.go:1196] Started watching for new ooms in manager
Sep 19 11:33:37.813478 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.813467    2584 manager.go:349] Starting recovery of all containers
Sep 19 11:33:37.814672 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:37.814648    2584 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-209.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 19 11:33:37.814771 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:37.814750    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 19 11:33:37.835318 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.835271    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hk5mg"
Sep 19 11:33:37.836593 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.836564    2584 manager.go:354] Recovery completed
Sep 19 11:33:37.840823 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.840792    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hk5mg"
Sep 19 11:33:37.842015 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.841998    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 11:33:37.843824 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.843806    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 11:33:37.843900 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.843840    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 11:33:37.843900 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.843851    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasSufficientPID"
Sep 19 11:33:37.844467 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.844446    2584 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 19 11:33:37.844467 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.844465    2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 19 11:33:37.844569 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.844479    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 11:33:37.844569 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.844485    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 11:33:37.845284 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.845267    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 11:33:37.845368 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.845304    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 11:33:37.845368 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.845317    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasSufficientPID"
Sep 19 11:33:37.846621 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.846604    2584 policy_none.go:49] "None policy: Start"
Sep 19 11:33:37.846621 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.846624    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 19 11:33:37.846733 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.846633    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 19 11:33:37.848012 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:37.846809    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-209.ec2.internal.18d6b5a3f9d360ff  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-209.ec2.internal,UID:ip-10-0-133-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-209.ec2.internal,},FirstTimestamp:2026-09-19 11:33:37.802027263 +0000 UTC m=+2.002057055,LastTimestamp:2026-09-19 11:33:37.802027263 +0000 UTC m=+2.002057055,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-209.ec2.internal,}"
Sep 19 11:33:37.848637 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.848625    2584 policy_none.go:47] "Start"
Sep 19 11:33:37.891661 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.891639    2584 manager.go:337] "Starting Device Plugin manager"
Sep 19 11:33:37.904972 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:37.891689    2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 19 11:33:37.904972 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.891702    2584 server.go:87] "Starting device plugin registration server"
Sep 19 11:33:37.904972 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.891983    2584 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 19 11:33:37.904972 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.892006    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 19 11:33:37.904972 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.892100    2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 19 11:33:37.904972 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.892220    2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 19 11:33:37.904972 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.892229    2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 19 11:33:37.904972 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:37.893528    2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 19 11:33:37.904972 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:37.893570    2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-209.ec2.internal\" not found"
Sep 19 11:33:37.904972 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.895055    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 19 11:33:37.904972 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.896446    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 19 11:33:37.904972 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.896485    2584 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 19 11:33:37.904972 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.896507    2584 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 19 11:33:37.904972 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:37.896554    2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 19 11:33:37.904972 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.898272    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 11:33:37.993194 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.993122    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 11:33:37.994127 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.994108    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 11:33:37.994201 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.994143    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 11:33:37.994201 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.994158    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasSufficientPID"
Sep 19 11:33:37.994201 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.994199    2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-209.ec2.internal"
Sep 19 11:33:37.997321 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.997300    2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-209.ec2.internal"]
Sep 19 11:33:37.997383 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.997374    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 11:33:37.998807 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.998790    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 11:33:37.998874 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.998822    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 11:33:37.998874 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:37.998835    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasSufficientPID"
Sep 19 11:33:38.001062 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.001047    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 11:33:38.001182 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.001163    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal"
Sep 19 11:33:38.001218 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.001194    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 11:33:38.002662 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.002647    2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-209.ec2.internal"
Sep 19 11:33:38.002704 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:38.002669    2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-209.ec2.internal\": node \"ip-10-0-133-209.ec2.internal\" not found"
Sep 19 11:33:38.008164 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.008146    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 11:33:38.008164 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.008174    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 11:33:38.008303 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.008186    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasSufficientPID"
Sep 19 11:33:38.008303 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.008221    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 11:33:38.008303 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.008241    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 11:33:38.008303 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.008250    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasSufficientPID"
Sep 19 11:33:38.010403 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.010389    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.ec2.internal"
Sep 19 11:33:38.010451 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.010421    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 11:33:38.011217 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.011194    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 11:33:38.011322 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.011221    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 11:33:38.011322 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.011239    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeHasSufficientPID"
Sep 19 11:33:38.013748 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.013729    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/49bc127c496c8981768be48e6fe5a22e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-209.ec2.internal\" (UID: \"49bc127c496c8981768be48e6fe5a22e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.ec2.internal"
Sep 19 11:33:38.013834 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.013753    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/58dbfd7134990f018404ee63672dd972-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal\" (UID: \"58dbfd7134990f018404ee63672dd972\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal"
Sep 19 11:33:38.013834 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.013802    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58dbfd7134990f018404ee63672dd972-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal\" (UID: \"58dbfd7134990f018404ee63672dd972\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal"
Sep 19 11:33:38.030650 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:38.030623    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-209.ec2.internal\" not found" node="ip-10-0-133-209.ec2.internal"
Sep 19 11:33:38.034901 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:38.034879    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-209.ec2.internal\" not found" node="ip-10-0-133-209.ec2.internal"
Sep 19 11:33:38.045940 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:38.045905    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-209.ec2.internal\" not found"
Sep 19 11:33:38.114274 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.114245    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/49bc127c496c8981768be48e6fe5a22e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-209.ec2.internal\" (UID: \"49bc127c496c8981768be48e6fe5a22e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.ec2.internal"
Sep 19 11:33:38.114393 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.114280    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/58dbfd7134990f018404ee63672dd972-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal\" (UID: \"58dbfd7134990f018404ee63672dd972\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal"
Sep 19 11:33:38.114393 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.114298    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58dbfd7134990f018404ee63672dd972-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal\" (UID: \"58dbfd7134990f018404ee63672dd972\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal"
Sep 19 11:33:38.114393 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.114332    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/49bc127c496c8981768be48e6fe5a22e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-209.ec2.internal\" (UID: \"49bc127c496c8981768be48e6fe5a22e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.ec2.internal"
Sep 19 11:33:38.114393 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.114351    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58dbfd7134990f018404ee63672dd972-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal\" (UID: \"58dbfd7134990f018404ee63672dd972\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal"
Sep 19 11:33:38.114393 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.114353    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/58dbfd7134990f018404ee63672dd972-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal\" (UID: \"58dbfd7134990f018404ee63672dd972\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal"
Sep 19 11:33:38.146352 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:38.146318    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-209.ec2.internal\" not found"
Sep 19 11:33:38.246935 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:38.246833    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-209.ec2.internal\" not found"
Sep 19 11:33:38.333785 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.333753    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal"
Sep 19 11:33:38.340123 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.340095    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.ec2.internal"
Sep 19 11:33:38.347082 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:38.346943    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-209.ec2.internal\" not found"
Sep 19 11:33:38.447550 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:38.447501    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-209.ec2.internal\" not found"
Sep 19 11:33:38.548029 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:38.547959    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-209.ec2.internal\" not found"
Sep 19 11:33:38.648498 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:38.648455    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-209.ec2.internal\" not found"
Sep 19 11:33:38.659420 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.659392    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 11:33:38.705310 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.705278    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 11:33:38.749180 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:38.749148    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-209.ec2.internal\" not found"
Sep 19 11:33:38.750248 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.750228    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 19 11:33:38.750418 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.750399    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 19 11:33:38.750469 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.750406    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 19 11:33:38.750469 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.750406    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 19 11:33:38.846221 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.846163    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-18 11:28:37 +0000 UTC" deadline="2028-03-31 05:57:49.394118024 +0000 UTC"
Sep 19 11:33:38.846221 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.846217    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13410h24m10.547905734s"
Sep 19 11:33:38.849265 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:38.849240    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-209.ec2.internal\" not found"
Sep 19 11:33:38.912940 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:38.912876    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49bc127c496c8981768be48e6fe5a22e.slice/crio-450e44b9e12d3b7f88ebc9b2127b7f0778a4308ce7168ffbb15ea804832e520f WatchSource:0}: Error finding container 450e44b9e12d3b7f88ebc9b2127b7f0778a4308ce7168ffbb15ea804832e520f: Status 404 returned error can't find the container with id 450e44b9e12d3b7f88ebc9b2127b7f0778a4308ce7168ffbb15ea804832e520f
Sep 19 11:33:38.913483 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:38.913451    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58dbfd7134990f018404ee63672dd972.slice/crio-610eedd9e91d958a3f6e1a083acfc1e983603ffae83f27e65a445036ecb26eb4 WatchSource:0}: Error finding container 610eedd9e91d958a3f6e1a083acfc1e983603ffae83f27e65a445036ecb26eb4: Status 404 returned error can't find the container with id 610eedd9e91d958a3f6e1a083acfc1e983603ffae83f27e65a445036ecb26eb4
Sep 19 11:33:38.918688 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.918666    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 11:33:38.944599 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:38.944559    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 11:33:39.012572 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.012538    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal"
Sep 19 11:33:39.024575 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.024545    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 11:33:39.025563 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.025548    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.ec2.internal"
Sep 19 11:33:39.033306 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.033290    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 11:33:39.668569 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.668532    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 11:33:39.791789 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.791751    2584 apiserver.go:52] "Watching apiserver"
Sep 19 11:33:39.801310 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.801273    2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 19 11:33:39.801714 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.801677    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-6ts8x","openshift-dns/node-resolver-vc5vt","openshift-multus/network-metrics-daemon-pgph6","openshift-network-operator/iptables-alerter-ndsnt","openshift-ovn-kubernetes/ovnkube-node-6th98","kube-system/global-pull-secret-syncer-dpgkd","kube-system/konnectivity-agent-dktw9","kube-system/kube-apiserver-proxy-ip-10-0-133-209.ec2.internal","openshift-image-registry/node-ca-rsnfw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal","openshift-multus/multus-6pbcp","openshift-multus/multus-additional-cni-plugins-ntx42","openshift-network-diagnostics/network-check-target-m2dhj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"]
Sep 19 11:33:39.804398 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.804375    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dktw9"
Sep 19 11:33:39.806547 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.806524    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:33:39.806660 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:39.806631    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2dhj" podUID="d0c272c9-c396-486e-b63b-6726cf1321d8"
Sep 19 11:33:39.807178 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.807158    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zxfrx\""
Sep 19 11:33:39.807287 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.807161    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 19 11:33:39.807349 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.807335    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 19 11:33:39.808693 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.808673    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:39.808817 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:39.808759    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgph6" podUID="83707003-2551-42b5-8943-8243dd6daa80"
Sep 19 11:33:39.810948 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.810932    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ndsnt"
Sep 19 11:33:39.813196 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.813113    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 19 11:33:39.814186 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.813365    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.814186 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.814053    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 19 11:33:39.814186 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.814085    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-69z9h\""
Sep 19 11:33:39.814186 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.814103    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 19 11:33:39.814409 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.814376    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 19 11:33:39.815824 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.815689    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:39.815824 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:39.815752    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpgkd" podUID="f96896a2-5494-4bc6-81c9-154860ee5f6b"
Sep 19 11:33:39.816008 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.815992    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 19 11:33:39.816500 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.816444    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 19 11:33:39.816981 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.816822    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 19 11:33:39.817358 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.817339    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 19 11:33:39.817456 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.817379    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dw6nj\""
Sep 19 11:33:39.818023 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.817998    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rsnfw"
Sep 19 11:33:39.818931 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.818526    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 19 11:33:39.818931 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.818717    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 19 11:33:39.821231 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.820382    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 19 11:33:39.821231 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.820539    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7bcvk\""
Sep 19 11:33:39.821231 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.820650    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 19 11:33:39.821231 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.820786    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:39.821231 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.820947    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 19 11:33:39.822948 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.822900    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 19 11:33:39.823299 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.823280    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vc5vt"
Sep 19 11:33:39.823366 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.823336    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hnchx\""
Sep 19 11:33:39.823556 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.823543    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 19 11:33:39.824166 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824147    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-cni-netd\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.824248 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824181    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs\") pod \"network-metrics-daemon-pgph6\" (UID: \"83707003-2551-42b5-8943-8243dd6daa80\") " pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:39.824248 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824205    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mh4x\" (UniqueName: \"kubernetes.io/projected/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-kube-api-access-4mh4x\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.824248 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824233    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7wwl\" (UniqueName: \"kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl\") pod \"network-check-target-m2dhj\" (UID: \"d0c272c9-c396-486e-b63b-6726cf1321d8\") " pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:33:39.824400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824256    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-run-systemd\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.824400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824277    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-run-ovn\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.824400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824297    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-cni-bin\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.824400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824318    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g7gp\" (UniqueName: \"kubernetes.io/projected/e84ed163-05a3-45ce-95b9-b5236328e3e3-kube-api-access-4g7gp\") pod \"iptables-alerter-ndsnt\" (UID: \"e84ed163-05a3-45ce-95b9-b5236328e3e3\") " pod="openshift-network-operator/iptables-alerter-ndsnt"
Sep 19 11:33:39.824400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824339    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-slash\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.824400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824360    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-run-openvswitch\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.824400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824377    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-log-socket\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.824749 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824417    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.824749 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824442    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-ovn-node-metrics-cert\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.824749 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824458    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-kubelet\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.824749 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824471    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ee2c4ee6-287a-4768-9cba-cb8729cef657-konnectivity-ca\") pod \"konnectivity-agent-dktw9\" (UID: \"ee2c4ee6-287a-4768-9cba-cb8729cef657\") " pod="kube-system/konnectivity-agent-dktw9"
Sep 19 11:33:39.824749 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824505    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-var-lib-openvswitch\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.824749 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824536    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-ovnkube-config\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.824749 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824551    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-ovnkube-script-lib\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.824749 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824567    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-etc-openvswitch\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.825239 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824593    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-env-overrides\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.825239 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824859    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ee2c4ee6-287a-4768-9cba-cb8729cef657-agent-certs\") pod \"konnectivity-agent-dktw9\" (UID: \"ee2c4ee6-287a-4768-9cba-cb8729cef657\") " pod="kube-system/konnectivity-agent-dktw9"
Sep 19 11:33:39.825239 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824901    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-run-netns\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.825239 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.824948    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k4dv\" (UniqueName: \"kubernetes.io/projected/83707003-2551-42b5-8943-8243dd6daa80-kube-api-access-4k4dv\") pod \"network-metrics-daemon-pgph6\" (UID: \"83707003-2551-42b5-8943-8243dd6daa80\") " pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:39.825239 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.825030    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e84ed163-05a3-45ce-95b9-b5236328e3e3-iptables-alerter-script\") pod \"iptables-alerter-ndsnt\" (UID: \"e84ed163-05a3-45ce-95b9-b5236328e3e3\") " pod="openshift-network-operator/iptables-alerter-ndsnt"
Sep 19 11:33:39.825239 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.825047    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e84ed163-05a3-45ce-95b9-b5236328e3e3-host-slash\") pod \"iptables-alerter-ndsnt\" (UID: \"e84ed163-05a3-45ce-95b9-b5236328e3e3\") " pod="openshift-network-operator/iptables-alerter-ndsnt"
Sep 19 11:33:39.825239 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.825074    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-node-log\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.825239 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.825097    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-run-ovn-kubernetes\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.825239 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.825160    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-systemd-units\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.826971 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.826950    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 19 11:33:39.827420 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.827398    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:39.828031 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.828009    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pb4jp\""
Sep 19 11:33:39.828495 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.828471    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 19 11:33:39.829355 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.829327    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 11:33:39.829821 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.829801    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 19 11:33:39.829985 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.829967    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 19 11:33:39.830259 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.830244    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5zwv7\""
Sep 19 11:33:39.830334 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.830262    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 19 11:33:39.830928 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.830896    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:39.833958 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.833605    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 19 11:33:39.833958 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.833903    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 19 11:33:39.834093 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.833958    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4pmfh\""
Sep 19 11:33:39.834093 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.834026    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 19 11:33:39.834205 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.834190    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 19 11:33:39.835599 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.835583    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:39.837755 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.837686    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 19 11:33:39.837860 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.837852    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 19 11:33:39.838064 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.837999    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9vs4j\""
Sep 19 11:33:39.850692 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.850561    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8f6ph"
Sep 19 11:33:39.858702 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.858682    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8f6ph"
Sep 19 11:33:39.902604 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.902534    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal" event={"ID":"58dbfd7134990f018404ee63672dd972","Type":"ContainerStarted","Data":"610eedd9e91d958a3f6e1a083acfc1e983603ffae83f27e65a445036ecb26eb4"}
Sep 19 11:33:39.903852 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.903798    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.ec2.internal" event={"ID":"49bc127c496c8981768be48e6fe5a22e","Type":"ContainerStarted","Data":"450e44b9e12d3b7f88ebc9b2127b7f0778a4308ce7168ffbb15ea804832e520f"}
Sep 19 11:33:39.914039 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.914017    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 19 11:33:39.926231 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926159    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-kubelet\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.926231 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926199    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2e7da35-ed65-4cd7-8d44-9b25ff53fba2-host\") pod \"node-ca-rsnfw\" (UID: \"b2e7da35-ed65-4cd7-8d44-9b25ff53fba2\") " pod="openshift-image-registry/node-ca-rsnfw"
Sep 19 11:33:39.926231 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926220    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-tuned\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:39.926446 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926240    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b566375-2b61-4d12-b35d-6c019ef72fb1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:39.926446 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926261    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-host-var-lib-cni-bin\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:39.926446 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926287    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-hostroot\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:39.926446 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926308    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-multus-daemon-config\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:39.926446 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926327    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-host-run-multus-certs\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:39.926446 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926354    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-var-lib-openvswitch\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.926446 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926378    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-ovnkube-config\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.926446 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926400    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-modprobe-d\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:39.926446 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926425    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dd0a400-b99d-4d02-b42c-f05d4f40a484-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:39.926850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926447    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-etc-openvswitch\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.926850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926471    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-env-overrides\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.926850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926495    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f96896a2-5494-4bc6-81c9-154860ee5f6b-dbus\") pod \"global-pull-secret-syncer-dpgkd\" (UID: \"f96896a2-5494-4bc6-81c9-154860ee5f6b\") " pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:39.926850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926517    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-sys\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:39.926850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926551    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e84ed163-05a3-45ce-95b9-b5236328e3e3-host-slash\") pod \"iptables-alerter-ndsnt\" (UID: \"e84ed163-05a3-45ce-95b9-b5236328e3e3\") " pod="openshift-network-operator/iptables-alerter-ndsnt"
Sep 19 11:33:39.926850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926577    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/869b7a50-52b6-4167-aa2d-f5152f2c8eb4-hosts-file\") pod \"node-resolver-vc5vt\" (UID: \"869b7a50-52b6-4167-aa2d-f5152f2c8eb4\") " pod="openshift-dns/node-resolver-vc5vt"
Sep 19 11:33:39.926850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926599    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns452\" (UniqueName: \"kubernetes.io/projected/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-kube-api-access-ns452\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:39.926850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926628    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dd0a400-b99d-4d02-b42c-f05d4f40a484-cnibin\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:39.926850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926650    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-cni-netd\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.926850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926672    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-var-lib-kubelet\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:39.926850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926694    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-multus-socket-dir-parent\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:39.926850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926728    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbsgc\" (UniqueName: \"kubernetes.io/projected/8dd0a400-b99d-4d02-b42c-f05d4f40a484-kube-api-access-rbsgc\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:39.926850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926757    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs\") pod \"network-metrics-daemon-pgph6\" (UID: \"83707003-2551-42b5-8943-8243dd6daa80\") " pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:39.926850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926783    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-cnibin\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:39.926850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926815    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-run-systemd\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.926850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926841    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-run-ovn\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.927682 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926864    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-cni-bin\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.927682 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926885    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-multus-cni-dir\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:39.927682 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926933    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8dd0a400-b99d-4d02-b42c-f05d4f40a484-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:39.927682 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.926985    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g7gp\" (UniqueName: \"kubernetes.io/projected/e84ed163-05a3-45ce-95b9-b5236328e3e3-kube-api-access-4g7gp\") pod \"iptables-alerter-ndsnt\" (UID: \"e84ed163-05a3-45ce-95b9-b5236328e3e3\") " pod="openshift-network-operator/iptables-alerter-ndsnt"
Sep 19 11:33:39.927682 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927012    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-run-openvswitch\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.927682 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927040    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.927682 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927093    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret\") pod \"global-pull-secret-syncer-dpgkd\" (UID: \"f96896a2-5494-4bc6-81c9-154860ee5f6b\") " pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:39.927682 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927119    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8tq6\" (UniqueName: \"kubernetes.io/projected/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-kube-api-access-n8tq6\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:39.927682 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927144    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8dd0a400-b99d-4d02-b42c-f05d4f40a484-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:39.927682 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927179    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ee2c4ee6-287a-4768-9cba-cb8729cef657-konnectivity-ca\") pod \"konnectivity-agent-dktw9\" (UID: \"ee2c4ee6-287a-4768-9cba-cb8729cef657\") " pod="kube-system/konnectivity-agent-dktw9"
Sep 19 11:33:39.927682 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927202    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/869b7a50-52b6-4167-aa2d-f5152f2c8eb4-tmp-dir\") pod \"node-resolver-vc5vt\" (UID: \"869b7a50-52b6-4167-aa2d-f5152f2c8eb4\") " pod="openshift-dns/node-resolver-vc5vt"
Sep 19 11:33:39.927682 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927225    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-sysctl-d\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:39.927682 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927250    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b566375-2b61-4d12-b35d-6c019ef72fb1-etc-selinux\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:39.927682 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927273    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-system-cni-dir\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:39.927682 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927296    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-ovnkube-script-lib\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.927682 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927319    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b2e7da35-ed65-4cd7-8d44-9b25ff53fba2-serviceca\") pod \"node-ca-rsnfw\" (UID: \"b2e7da35-ed65-4cd7-8d44-9b25ff53fba2\") " pod="openshift-image-registry/node-ca-rsnfw"
Sep 19 11:33:39.928463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927341    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-kubernetes\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:39.928463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927366    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-systemd\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:39.928463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927390    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b566375-2b61-4d12-b35d-6c019ef72fb1-socket-dir\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:39.928463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927418    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-etc-kubernetes\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:39.928463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927442    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dd0a400-b99d-4d02-b42c-f05d4f40a484-system-cni-dir\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:39.928463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927467    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ee2c4ee6-287a-4768-9cba-cb8729cef657-agent-certs\") pod \"konnectivity-agent-dktw9\" (UID: \"ee2c4ee6-287a-4768-9cba-cb8729cef657\") " pod="kube-system/konnectivity-agent-dktw9"
Sep 19 11:33:39.928463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927492    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-sysctl-conf\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:39.928463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927516    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b566375-2b61-4d12-b35d-6c019ef72fb1-registration-dir\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:39.928463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927539    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b566375-2b61-4d12-b35d-6c019ef72fb1-device-dir\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:39.928463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927563    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h2km\" (UniqueName: \"kubernetes.io/projected/2b566375-2b61-4d12-b35d-6c019ef72fb1-kube-api-access-5h2km\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:39.928463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927589    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-host-var-lib-kubelet\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:39.928463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927619    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-run-netns\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.928463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927647    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k4dv\" (UniqueName: \"kubernetes.io/projected/83707003-2551-42b5-8943-8243dd6daa80-kube-api-access-4k4dv\") pod \"network-metrics-daemon-pgph6\" (UID: \"83707003-2551-42b5-8943-8243dd6daa80\") " pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:39.928463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927674    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e84ed163-05a3-45ce-95b9-b5236328e3e3-iptables-alerter-script\") pod \"iptables-alerter-ndsnt\" (UID: \"e84ed163-05a3-45ce-95b9-b5236328e3e3\") " pod="openshift-network-operator/iptables-alerter-ndsnt"
Sep 19 11:33:39.928463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927697    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-node-log\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.928463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927721    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-run-ovn-kubernetes\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.929187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927747    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs9dl\" (UniqueName: \"kubernetes.io/projected/b2e7da35-ed65-4cd7-8d44-9b25ff53fba2-kube-api-access-xs9dl\") pod \"node-ca-rsnfw\" (UID: \"b2e7da35-ed65-4cd7-8d44-9b25ff53fba2\") " pod="openshift-image-registry/node-ca-rsnfw"
Sep 19 11:33:39.929187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927770    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-host\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:39.929187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927794    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-host-run-k8s-cni-cncf-io\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:39.929187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927820    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-systemd-units\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.929187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927857    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-tmp\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:39.929187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927882    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-host-run-netns\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:39.929187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927907    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8dd0a400-b99d-4d02-b42c-f05d4f40a484-cni-binary-copy\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:39.929187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927957    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mh4x\" (UniqueName: \"kubernetes.io/projected/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-kube-api-access-4mh4x\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.929187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.927991    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f96896a2-5494-4bc6-81c9-154860ee5f6b-kubelet-config\") pod \"global-pull-secret-syncer-dpgkd\" (UID: \"f96896a2-5494-4bc6-81c9-154860ee5f6b\") " pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:39.929187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.928014    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-run\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:39.929187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.928037    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-lib-modules\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:39.929187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.928060    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b566375-2b61-4d12-b35d-6c019ef72fb1-sys-fs\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:39.929187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.928089    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7wwl\" (UniqueName: \"kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl\") pod \"network-check-target-m2dhj\" (UID: \"d0c272c9-c396-486e-b63b-6726cf1321d8\") " pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:33:39.929187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.928114    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kctnn\" (UniqueName: \"kubernetes.io/projected/869b7a50-52b6-4167-aa2d-f5152f2c8eb4-kube-api-access-kctnn\") pod \"node-resolver-vc5vt\" (UID: \"869b7a50-52b6-4167-aa2d-f5152f2c8eb4\") " pod="openshift-dns/node-resolver-vc5vt"
Sep 19 11:33:39.929187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.928138    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-sysconfig\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:39.929187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.928162    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-cni-binary-copy\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:39.929187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.928187    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-host-var-lib-cni-multus\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:39.929985 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.928213    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-slash\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.929985 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.928258    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-log-socket\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.929985 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.928283    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-ovn-node-metrics-cert\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.929985 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.928308    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-multus-conf-dir\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:39.929985 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.928404    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-kubelet\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.929985 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.928518    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-var-lib-openvswitch\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.929985 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.929170    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-ovnkube-config\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.929985 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.929283    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-etc-openvswitch\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.929985 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.929622    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-env-overrides\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.929985 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.929702    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e84ed163-05a3-45ce-95b9-b5236328e3e3-host-slash\") pod \"iptables-alerter-ndsnt\" (UID: \"e84ed163-05a3-45ce-95b9-b5236328e3e3\") " pod="openshift-network-operator/iptables-alerter-ndsnt"
Sep 19 11:33:39.929985 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.929776    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-cni-netd\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.929985 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:39.929886    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 11:33:39.929985 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:39.929972    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs podName:83707003-2551-42b5-8943-8243dd6daa80 nodeName:}" failed. No retries permitted until 2026-09-19 11:33:40.429948289 +0000 UTC m=+4.629978090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs") pod "network-metrics-daemon-pgph6" (UID: "83707003-2551-42b5-8943-8243dd6daa80") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 11:33:39.930563 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.930211    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-run-systemd\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.930563 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.930261    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-run-ovn\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.930563 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.930303    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-cni-bin\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.930709 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.930570    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-run-openvswitch\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.930709 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.930620    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.931146 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.931072    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ee2c4ee6-287a-4768-9cba-cb8729cef657-konnectivity-ca\") pod \"konnectivity-agent-dktw9\" (UID: \"ee2c4ee6-287a-4768-9cba-cb8729cef657\") " pod="kube-system/konnectivity-agent-dktw9"
Sep 19 11:33:39.931539 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.931515    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-ovnkube-script-lib\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.931858 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.931835    2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 19 11:33:39.932936 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.932161    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-node-log\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.932936 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.932302    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-run-ovn-kubernetes\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.932936 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.932350    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-systemd-units\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.932936 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.932379    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-slash\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.932936 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.932538    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-host-run-netns\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.932936 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.932593    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e84ed163-05a3-45ce-95b9-b5236328e3e3-iptables-alerter-script\") pod \"iptables-alerter-ndsnt\" (UID: \"e84ed163-05a3-45ce-95b9-b5236328e3e3\") " pod="openshift-network-operator/iptables-alerter-ndsnt"
Sep 19 11:33:39.932936 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.932725    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-log-socket\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.936376 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.936353    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ee2c4ee6-287a-4768-9cba-cb8729cef657-agent-certs\") pod \"konnectivity-agent-dktw9\" (UID: \"ee2c4ee6-287a-4768-9cba-cb8729cef657\") " pod="kube-system/konnectivity-agent-dktw9"
Sep 19 11:33:39.936959 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.936935    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-ovn-node-metrics-cert\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:39.939924 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:39.939887    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 11:33:39.940023 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:39.939963    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 11:33:39.940023 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:39.939979    2584 projected.go:196] Error preparing data for projected volume kube-api-access-w7wwl for pod openshift-network-diagnostics/network-check-target-m2dhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 11:33:39.940123 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:39.940039    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl podName:d0c272c9-c396-486e-b63b-6726cf1321d8 nodeName:}" failed. No retries permitted until 2026-09-19 11:33:40.440022277 +0000 UTC m=+4.640052075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w7wwl" (UniqueName: "kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl") pod "network-check-target-m2dhj" (UID: "d0c272c9-c396-486e-b63b-6726cf1321d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 11:33:39.942107 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.942038    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g7gp\" (UniqueName: \"kubernetes.io/projected/e84ed163-05a3-45ce-95b9-b5236328e3e3-kube-api-access-4g7gp\") pod \"iptables-alerter-ndsnt\" (UID: \"e84ed163-05a3-45ce-95b9-b5236328e3e3\") " pod="openshift-network-operator/iptables-alerter-ndsnt"
Sep 19 11:33:39.942408 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.942386    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k4dv\" (UniqueName: \"kubernetes.io/projected/83707003-2551-42b5-8943-8243dd6daa80-kube-api-access-4k4dv\") pod \"network-metrics-daemon-pgph6\" (UID: \"83707003-2551-42b5-8943-8243dd6daa80\") " pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:39.942704 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:39.942685    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mh4x\" (UniqueName: \"kubernetes.io/projected/2fed27e0-833f-49a8-b11e-8f61df6e6bd5-kube-api-access-4mh4x\") pod \"ovnkube-node-6th98\" (UID: \"2fed27e0-833f-49a8-b11e-8f61df6e6bd5\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:40.029422 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029380    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-cnibin\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.029422 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029427    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-multus-cni-dir\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.029690 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029451    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8dd0a400-b99d-4d02-b42c-f05d4f40a484-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:40.029690 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029535    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-cnibin\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.029690 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029571    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret\") pod \"global-pull-secret-syncer-dpgkd\" (UID: \"f96896a2-5494-4bc6-81c9-154860ee5f6b\") " pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:40.029690 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029619    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8tq6\" (UniqueName: \"kubernetes.io/projected/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-kube-api-access-n8tq6\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.029690 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029644    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8dd0a400-b99d-4d02-b42c-f05d4f40a484-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:40.029690 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029674    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/869b7a50-52b6-4167-aa2d-f5152f2c8eb4-tmp-dir\") pod \"node-resolver-vc5vt\" (UID: \"869b7a50-52b6-4167-aa2d-f5152f2c8eb4\") " pod="openshift-dns/node-resolver-vc5vt"
Sep 19 11:33:40.029997 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029697    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-sysctl-d\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.029997 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029719    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b566375-2b61-4d12-b35d-6c019ef72fb1-etc-selinux\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:40.029997 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:40.029735    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 11:33:40.029997 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:40.029799    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret podName:f96896a2-5494-4bc6-81c9-154860ee5f6b nodeName:}" failed. No retries permitted until 2026-09-19 11:33:40.529780948 +0000 UTC m=+4.729810744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret") pod "global-pull-secret-syncer-dpgkd" (UID: "f96896a2-5494-4bc6-81c9-154860ee5f6b") : object "kube-system"/"original-pull-secret" not registered
Sep 19 11:33:40.029997 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029740    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-system-cni-dir\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.029997 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029833    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-sysctl-d\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.029997 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029834    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b2e7da35-ed65-4cd7-8d44-9b25ff53fba2-serviceca\") pod \"node-ca-rsnfw\" (UID: \"b2e7da35-ed65-4cd7-8d44-9b25ff53fba2\") " pod="openshift-image-registry/node-ca-rsnfw"
Sep 19 11:33:40.029997 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029841    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b566375-2b61-4d12-b35d-6c019ef72fb1-etc-selinux\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:40.029997 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029863    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-kubernetes\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.029997 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029886    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-systemd\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.029997 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029927    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b566375-2b61-4d12-b35d-6c019ef72fb1-socket-dir\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:40.029997 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029672    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-multus-cni-dir\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.029997 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029950    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-etc-kubernetes\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.029997 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029983    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-etc-kubernetes\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.029997 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.029989    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dd0a400-b99d-4d02-b42c-f05d4f40a484-system-cni-dir\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:40.030713 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030019    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-sysctl-conf\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.030713 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030027    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-systemd\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.030713 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030044    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b566375-2b61-4d12-b35d-6c019ef72fb1-registration-dir\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:40.030713 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030062    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/869b7a50-52b6-4167-aa2d-f5152f2c8eb4-tmp-dir\") pod \"node-resolver-vc5vt\" (UID: \"869b7a50-52b6-4167-aa2d-f5152f2c8eb4\") " pod="openshift-dns/node-resolver-vc5vt"
Sep 19 11:33:40.030713 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030094    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dd0a400-b99d-4d02-b42c-f05d4f40a484-system-cni-dir\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:40.030713 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030068    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b566375-2b61-4d12-b35d-6c019ef72fb1-device-dir\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:40.030713 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030108    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b566375-2b61-4d12-b35d-6c019ef72fb1-device-dir\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:40.030713 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030127    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b566375-2b61-4d12-b35d-6c019ef72fb1-socket-dir\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:40.030713 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030136    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2km\" (UniqueName: \"kubernetes.io/projected/2b566375-2b61-4d12-b35d-6c019ef72fb1-kube-api-access-5h2km\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:40.030713 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030155    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b566375-2b61-4d12-b35d-6c019ef72fb1-registration-dir\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:40.030713 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030157    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-kubernetes\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.030713 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030168    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-host-var-lib-kubelet\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.030713 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030125    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8dd0a400-b99d-4d02-b42c-f05d4f40a484-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:40.030713 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030178    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-system-cni-dir\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.030713 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030200    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs9dl\" (UniqueName: \"kubernetes.io/projected/b2e7da35-ed65-4cd7-8d44-9b25ff53fba2-kube-api-access-xs9dl\") pod \"node-ca-rsnfw\" (UID: \"b2e7da35-ed65-4cd7-8d44-9b25ff53fba2\") " pod="openshift-image-registry/node-ca-rsnfw"
Sep 19 11:33:40.030713 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030223    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-host\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.030713 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030224    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8dd0a400-b99d-4d02-b42c-f05d4f40a484-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:40.031436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030234    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-host-var-lib-kubelet\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.031436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030244    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-host-run-k8s-cni-cncf-io\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.031436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030256    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-sysctl-conf\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.031436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030266    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-tmp\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.031436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030289    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-host-run-netns\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.031436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030293    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-host-run-k8s-cni-cncf-io\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.031436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030311    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8dd0a400-b99d-4d02-b42c-f05d4f40a484-cni-binary-copy\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:40.031436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030326    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b2e7da35-ed65-4cd7-8d44-9b25ff53fba2-serviceca\") pod \"node-ca-rsnfw\" (UID: \"b2e7da35-ed65-4cd7-8d44-9b25ff53fba2\") " pod="openshift-image-registry/node-ca-rsnfw"
Sep 19 11:33:40.031436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030338    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f96896a2-5494-4bc6-81c9-154860ee5f6b-kubelet-config\") pod \"global-pull-secret-syncer-dpgkd\" (UID: \"f96896a2-5494-4bc6-81c9-154860ee5f6b\") " pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:40.031436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030346    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-host-run-netns\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.031436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030290    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-host\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.031436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030371    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-run\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.031436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030388    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f96896a2-5494-4bc6-81c9-154860ee5f6b-kubelet-config\") pod \"global-pull-secret-syncer-dpgkd\" (UID: \"f96896a2-5494-4bc6-81c9-154860ee5f6b\") " pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:40.031436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030407    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-lib-modules\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.031436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030409    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-run\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.031436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030434    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b566375-2b61-4d12-b35d-6c019ef72fb1-sys-fs\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:40.031436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030482    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kctnn\" (UniqueName: \"kubernetes.io/projected/869b7a50-52b6-4167-aa2d-f5152f2c8eb4-kube-api-access-kctnn\") pod \"node-resolver-vc5vt\" (UID: \"869b7a50-52b6-4167-aa2d-f5152f2c8eb4\") " pod="openshift-dns/node-resolver-vc5vt"
Sep 19 11:33:40.031436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030491    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-lib-modules\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.032017 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030503    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-sysconfig\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.032017 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030524    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-cni-binary-copy\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.032017 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030548    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-host-var-lib-cni-multus\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.032017 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030550    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b566375-2b61-4d12-b35d-6c019ef72fb1-sys-fs\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:40.032017 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030574    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-multus-conf-dir\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.032017 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030597    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2e7da35-ed65-4cd7-8d44-9b25ff53fba2-host\") pod \"node-ca-rsnfw\" (UID: \"b2e7da35-ed65-4cd7-8d44-9b25ff53fba2\") " pod="openshift-image-registry/node-ca-rsnfw"
Sep 19 11:33:40.032017 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030624    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-tuned\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.032017 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030637    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-multus-conf-dir\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.032017 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030649    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b566375-2b61-4d12-b35d-6c019ef72fb1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:40.032017 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030598    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-sysconfig\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.032017 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030671    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-host-var-lib-cni-bin\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.032017 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030648    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-host-var-lib-cni-multus\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.032017 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030722    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2e7da35-ed65-4cd7-8d44-9b25ff53fba2-host\") pod \"node-ca-rsnfw\" (UID: \"b2e7da35-ed65-4cd7-8d44-9b25ff53fba2\") " pod="openshift-image-registry/node-ca-rsnfw"
Sep 19 11:33:40.032017 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030722    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b566375-2b61-4d12-b35d-6c019ef72fb1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:40.032017 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030742    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-hostroot\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.032017 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030766    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-multus-daemon-config\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.032017 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030770    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-host-var-lib-cni-bin\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.032017 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030770    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8dd0a400-b99d-4d02-b42c-f05d4f40a484-cni-binary-copy\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:40.033060 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030781    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-host-run-multus-certs\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.033060 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030806    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-modprobe-d\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.033060 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030814    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-hostroot\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.033060 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030828    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dd0a400-b99d-4d02-b42c-f05d4f40a484-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:40.033060 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030844    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f96896a2-5494-4bc6-81c9-154860ee5f6b-dbus\") pod \"global-pull-secret-syncer-dpgkd\" (UID: \"f96896a2-5494-4bc6-81c9-154860ee5f6b\") " pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:40.033060 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030859    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-sys\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.033060 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030875    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/869b7a50-52b6-4167-aa2d-f5152f2c8eb4-hosts-file\") pod \"node-resolver-vc5vt\" (UID: \"869b7a50-52b6-4167-aa2d-f5152f2c8eb4\") " pod="openshift-dns/node-resolver-vc5vt"
Sep 19 11:33:40.033060 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030891    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns452\" (UniqueName: \"kubernetes.io/projected/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-kube-api-access-ns452\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.033060 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030927    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dd0a400-b99d-4d02-b42c-f05d4f40a484-cnibin\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:40.033060 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030950    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-var-lib-kubelet\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.033060 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030972    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-multus-socket-dir-parent\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.033060 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030994    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbsgc\" (UniqueName: \"kubernetes.io/projected/8dd0a400-b99d-4d02-b42c-f05d4f40a484-kube-api-access-rbsgc\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:40.033060 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.031024    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-modprobe-d\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.033060 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.031021    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dd0a400-b99d-4d02-b42c-f05d4f40a484-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:40.033060 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.031080    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/869b7a50-52b6-4167-aa2d-f5152f2c8eb4-hosts-file\") pod \"node-resolver-vc5vt\" (UID: \"869b7a50-52b6-4167-aa2d-f5152f2c8eb4\") " pod="openshift-dns/node-resolver-vc5vt"
Sep 19 11:33:40.033060 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.031084    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dd0a400-b99d-4d02-b42c-f05d4f40a484-cnibin\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:40.033060 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.031138    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f96896a2-5494-4bc6-81c9-154860ee5f6b-dbus\") pod \"global-pull-secret-syncer-dpgkd\" (UID: \"f96896a2-5494-4bc6-81c9-154860ee5f6b\") " pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:40.033060 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.031166    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-cni-binary-copy\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.033883 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.031187    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-sys\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.033883 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.030858    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-host-run-multus-certs\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.033883 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.031235    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-var-lib-kubelet\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.033883 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.031242    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-multus-socket-dir-parent\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.033883 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.031244    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-multus-daemon-config\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.033883 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.032751    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-tmp\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.034142 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.034127    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-etc-tuned\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.038562 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.038533    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h2km\" (UniqueName: \"kubernetes.io/projected/2b566375-2b61-4d12-b35d-6c019ef72fb1-kube-api-access-5h2km\") pod \"aws-ebs-csi-driver-node-9d9ml\" (UID: \"2b566375-2b61-4d12-b35d-6c019ef72fb1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:40.038682 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.038661    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8tq6\" (UniqueName: \"kubernetes.io/projected/b1fb8bd8-ce45-4e39-a7df-96f844fa2825-kube-api-access-n8tq6\") pod \"tuned-6ts8x\" (UID: \"b1fb8bd8-ce45-4e39-a7df-96f844fa2825\") " pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.038833 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.038808    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbsgc\" (UniqueName: \"kubernetes.io/projected/8dd0a400-b99d-4d02-b42c-f05d4f40a484-kube-api-access-rbsgc\") pod \"multus-additional-cni-plugins-ntx42\" (UID: \"8dd0a400-b99d-4d02-b42c-f05d4f40a484\") " pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:40.039311 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.039265    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kctnn\" (UniqueName: \"kubernetes.io/projected/869b7a50-52b6-4167-aa2d-f5152f2c8eb4-kube-api-access-kctnn\") pod \"node-resolver-vc5vt\" (UID: \"869b7a50-52b6-4167-aa2d-f5152f2c8eb4\") " pod="openshift-dns/node-resolver-vc5vt"
Sep 19 11:33:40.039538 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.039515    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs9dl\" (UniqueName: \"kubernetes.io/projected/b2e7da35-ed65-4cd7-8d44-9b25ff53fba2-kube-api-access-xs9dl\") pod \"node-ca-rsnfw\" (UID: \"b2e7da35-ed65-4cd7-8d44-9b25ff53fba2\") " pod="openshift-image-registry/node-ca-rsnfw"
Sep 19 11:33:40.039692 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.039675    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns452\" (UniqueName: \"kubernetes.io/projected/11fdca9a-6699-4e5e-b55e-78ca9308ed2e-kube-api-access-ns452\") pod \"multus-6pbcp\" (UID: \"11fdca9a-6699-4e5e-b55e-78ca9308ed2e\") " pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.116857 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.116820    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dktw9"
Sep 19 11:33:40.134281 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.134250    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ndsnt"
Sep 19 11:33:40.138226 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.138200    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:33:40.145029 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.145002    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rsnfw"
Sep 19 11:33:40.152650 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.152628    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6ts8x"
Sep 19 11:33:40.160004 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.159978    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vc5vt"
Sep 19 11:33:40.166006 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.165979    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml"
Sep 19 11:33:40.171690 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.171672    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6pbcp"
Sep 19 11:33:40.177263 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.177211    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ntx42"
Sep 19 11:33:40.433689 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.433599    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs\") pod \"network-metrics-daemon-pgph6\" (UID: \"83707003-2551-42b5-8943-8243dd6daa80\") " pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:40.433886 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:40.433765    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 11:33:40.433886 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:40.433836    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs podName:83707003-2551-42b5-8943-8243dd6daa80 nodeName:}" failed. No retries permitted until 2026-09-19 11:33:41.433818844 +0000 UTC m=+5.633848629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs") pod "network-metrics-daemon-pgph6" (UID: "83707003-2551-42b5-8943-8243dd6daa80") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 11:33:40.497431 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.497400    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 11:33:40.522359 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.522330    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 11:33:40.534653 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.534620    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7wwl\" (UniqueName: \"kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl\") pod \"network-check-target-m2dhj\" (UID: \"d0c272c9-c396-486e-b63b-6726cf1321d8\") " pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:33:40.534807 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.534676    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret\") pod \"global-pull-secret-syncer-dpgkd\" (UID: \"f96896a2-5494-4bc6-81c9-154860ee5f6b\") " pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:40.534807 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:40.534766    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 11:33:40.534807 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:40.534789    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 11:33:40.534905 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:40.534815    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 11:33:40.534905 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:40.534826    2584 projected.go:196] Error preparing data for projected volume kube-api-access-w7wwl for pod openshift-network-diagnostics/network-check-target-m2dhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 11:33:40.534905 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:40.534817    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret podName:f96896a2-5494-4bc6-81c9-154860ee5f6b nodeName:}" failed. No retries permitted until 2026-09-19 11:33:41.534804395 +0000 UTC m=+5.734834175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret") pod "global-pull-secret-syncer-dpgkd" (UID: "f96896a2-5494-4bc6-81c9-154860ee5f6b") : object "kube-system"/"original-pull-secret" not registered
Sep 19 11:33:40.534905 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:40.534894    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl podName:d0c272c9-c396-486e-b63b-6726cf1321d8 nodeName:}" failed. No retries permitted until 2026-09-19 11:33:41.534878013 +0000 UTC m=+5.734907793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w7wwl" (UniqueName: "kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl") pod "network-check-target-m2dhj" (UID: "d0c272c9-c396-486e-b63b-6726cf1321d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 11:33:40.831664 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:40.831638    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b566375_2b61_4d12_b35d_6c019ef72fb1.slice/crio-f83134899957f4ea6765d98341b00e9007231d3ca5559b82f13c9190db6c5082 WatchSource:0}: Error finding container f83134899957f4ea6765d98341b00e9007231d3ca5559b82f13c9190db6c5082: Status 404 returned error can't find the container with id f83134899957f4ea6765d98341b00e9007231d3ca5559b82f13c9190db6c5082
Sep 19 11:33:40.832591 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:40.832567    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11fdca9a_6699_4e5e_b55e_78ca9308ed2e.slice/crio-16138200df250024f7d9fc8c4b2656b3312814eaaff7c62da9911777b32c53ed WatchSource:0}: Error finding container 16138200df250024f7d9fc8c4b2656b3312814eaaff7c62da9911777b32c53ed: Status 404 returned error can't find the container with id 16138200df250024f7d9fc8c4b2656b3312814eaaff7c62da9911777b32c53ed
Sep 19 11:33:40.834062 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:40.833971    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84ed163_05a3_45ce_95b9_b5236328e3e3.slice/crio-eb0f1ffffd7bed224634d11cc55072c2a459f7d88fbaf2a30060a89765035bda WatchSource:0}: Error finding container eb0f1ffffd7bed224634d11cc55072c2a459f7d88fbaf2a30060a89765035bda: Status 404 returned error can't find the container with id eb0f1ffffd7bed224634d11cc55072c2a459f7d88fbaf2a30060a89765035bda
Sep 19 11:33:40.838471 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:40.838448    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fed27e0_833f_49a8_b11e_8f61df6e6bd5.slice/crio-2bfdad6e23233a232a4a25b6a552a5da64befaa25b790be62c133a1011e6e420 WatchSource:0}: Error finding container 2bfdad6e23233a232a4a25b6a552a5da64befaa25b790be62c133a1011e6e420: Status 404 returned error can't find the container with id 2bfdad6e23233a232a4a25b6a552a5da64befaa25b790be62c133a1011e6e420
Sep 19 11:33:40.839583 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:40.839562    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fb8bd8_ce45_4e39_a7df_96f844fa2825.slice/crio-549f91ab79f89161defa5ca35fb70dc7a8a2427ceaaee35666fdb7811d5fd9a3 WatchSource:0}: Error finding container 549f91ab79f89161defa5ca35fb70dc7a8a2427ceaaee35666fdb7811d5fd9a3: Status 404 returned error can't find the container with id 549f91ab79f89161defa5ca35fb70dc7a8a2427ceaaee35666fdb7811d5fd9a3
Sep 19 11:33:40.840685 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:40.840667    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2c4ee6_287a_4768_9cba_cb8729cef657.slice/crio-350a7f6b64cd37c470d86918d65dce479ce31fc84da65e73c1b80347185229b0 WatchSource:0}: Error finding container 350a7f6b64cd37c470d86918d65dce479ce31fc84da65e73c1b80347185229b0: Status 404 returned error can't find the container with id 350a7f6b64cd37c470d86918d65dce479ce31fc84da65e73c1b80347185229b0
Sep 19 11:33:40.841052 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:40.841030    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e7da35_ed65_4cd7_8d44_9b25ff53fba2.slice/crio-4754ded04d34dc9c5e26cfad841db03165a1fb37c893270537273cc2c547e211 WatchSource:0}: Error finding container 4754ded04d34dc9c5e26cfad841db03165a1fb37c893270537273cc2c547e211: Status 404 returned error can't find the container with id 4754ded04d34dc9c5e26cfad841db03165a1fb37c893270537273cc2c547e211
Sep 19 11:33:40.844716 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:40.843071    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869b7a50_52b6_4167_aa2d_f5152f2c8eb4.slice/crio-7a0313599e32468cf8cc5c193f37f6708f0ea34ad14e091de708c5548719f5e8 WatchSource:0}: Error finding container 7a0313599e32468cf8cc5c193f37f6708f0ea34ad14e091de708c5548719f5e8: Status 404 returned error can't find the container with id 7a0313599e32468cf8cc5c193f37f6708f0ea34ad14e091de708c5548719f5e8
Sep 19 11:33:40.845236 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:33:40.844841    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd0a400_b99d_4d02_b42c_f05d4f40a484.slice/crio-4fc35fb55c6773f9ff253108b9a9fabbfe827cc216f0637fe117d811fffeebc6 WatchSource:0}: Error finding container 4fc35fb55c6773f9ff253108b9a9fabbfe827cc216f0637fe117d811fffeebc6: Status 404 returned error can't find the container with id 4fc35fb55c6773f9ff253108b9a9fabbfe827cc216f0637fe117d811fffeebc6
Sep 19 11:33:40.862189 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.862141    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 11:28:39 +0000 UTC" deadline="2028-06-13 07:29:04.615802806 +0000 UTC"
Sep 19 11:33:40.862189 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.862189    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15187h55m23.753618075s"
Sep 19 11:33:40.897291 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.897258    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:33:40.897410 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:40.897368    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2dhj" podUID="d0c272c9-c396-486e-b63b-6726cf1321d8"
Sep 19 11:33:40.906869 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.906820    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rsnfw" event={"ID":"b2e7da35-ed65-4cd7-8d44-9b25ff53fba2","Type":"ContainerStarted","Data":"4754ded04d34dc9c5e26cfad841db03165a1fb37c893270537273cc2c547e211"}
Sep 19 11:33:40.907945 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.907899    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6ts8x" event={"ID":"b1fb8bd8-ce45-4e39-a7df-96f844fa2825","Type":"ContainerStarted","Data":"549f91ab79f89161defa5ca35fb70dc7a8a2427ceaaee35666fdb7811d5fd9a3"}
Sep 19 11:33:40.909112 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.909080    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6th98" event={"ID":"2fed27e0-833f-49a8-b11e-8f61df6e6bd5","Type":"ContainerStarted","Data":"2bfdad6e23233a232a4a25b6a552a5da64befaa25b790be62c133a1011e6e420"}
Sep 19 11:33:40.910258 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.910232    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ndsnt" event={"ID":"e84ed163-05a3-45ce-95b9-b5236328e3e3","Type":"ContainerStarted","Data":"eb0f1ffffd7bed224634d11cc55072c2a459f7d88fbaf2a30060a89765035bda"}
Sep 19 11:33:40.911434 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.911405    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntx42" event={"ID":"8dd0a400-b99d-4d02-b42c-f05d4f40a484","Type":"ContainerStarted","Data":"4fc35fb55c6773f9ff253108b9a9fabbfe827cc216f0637fe117d811fffeebc6"}
Sep 19 11:33:40.912530 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.912505    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vc5vt" event={"ID":"869b7a50-52b6-4167-aa2d-f5152f2c8eb4","Type":"ContainerStarted","Data":"7a0313599e32468cf8cc5c193f37f6708f0ea34ad14e091de708c5548719f5e8"}
Sep 19 11:33:40.913637 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.913615    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dktw9" event={"ID":"ee2c4ee6-287a-4768-9cba-cb8729cef657","Type":"ContainerStarted","Data":"350a7f6b64cd37c470d86918d65dce479ce31fc84da65e73c1b80347185229b0"}
Sep 19 11:33:40.914737 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.914709    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6pbcp" event={"ID":"11fdca9a-6699-4e5e-b55e-78ca9308ed2e","Type":"ContainerStarted","Data":"16138200df250024f7d9fc8c4b2656b3312814eaaff7c62da9911777b32c53ed"}
Sep 19 11:33:40.915626 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:40.915607    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml" event={"ID":"2b566375-2b61-4d12-b35d-6c019ef72fb1","Type":"ContainerStarted","Data":"f83134899957f4ea6765d98341b00e9007231d3ca5559b82f13c9190db6c5082"}
Sep 19 11:33:41.441495 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:41.441200    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs\") pod \"network-metrics-daemon-pgph6\" (UID: \"83707003-2551-42b5-8943-8243dd6daa80\") " pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:41.441495 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:41.441446    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 11:33:41.441721 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:41.441534    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs podName:83707003-2551-42b5-8943-8243dd6daa80 nodeName:}" failed. No retries permitted until 2026-09-19 11:33:43.441512443 +0000 UTC m=+7.641542235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs") pod "network-metrics-daemon-pgph6" (UID: "83707003-2551-42b5-8943-8243dd6daa80") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 11:33:41.542234 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:41.542069    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7wwl\" (UniqueName: \"kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl\") pod \"network-check-target-m2dhj\" (UID: \"d0c272c9-c396-486e-b63b-6726cf1321d8\") " pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:33:41.542234 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:41.542151    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret\") pod \"global-pull-secret-syncer-dpgkd\" (UID: \"f96896a2-5494-4bc6-81c9-154860ee5f6b\") " pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:41.542472 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:41.542286    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 11:33:41.542472 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:41.542352    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret podName:f96896a2-5494-4bc6-81c9-154860ee5f6b nodeName:}" failed. No retries permitted until 2026-09-19 11:33:43.542333779 +0000 UTC m=+7.742363580 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret") pod "global-pull-secret-syncer-dpgkd" (UID: "f96896a2-5494-4bc6-81c9-154860ee5f6b") : object "kube-system"/"original-pull-secret" not registered
Sep 19 11:33:41.542779 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:41.542758    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 11:33:41.542857 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:41.542787    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 11:33:41.542857 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:41.542801    2584 projected.go:196] Error preparing data for projected volume kube-api-access-w7wwl for pod openshift-network-diagnostics/network-check-target-m2dhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 11:33:41.542857 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:41.542845    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl podName:d0c272c9-c396-486e-b63b-6726cf1321d8 nodeName:}" failed. No retries permitted until 2026-09-19 11:33:43.542830522 +0000 UTC m=+7.742860308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w7wwl" (UniqueName: "kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl") pod "network-check-target-m2dhj" (UID: "d0c272c9-c396-486e-b63b-6726cf1321d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 11:33:41.864664 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:41.864618    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 11:28:39 +0000 UTC" deadline="2028-03-24 12:10:35.774254589 +0000 UTC"
Sep 19 11:33:41.864664 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:41.864662    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13248h36m53.909597989s"
Sep 19 11:33:41.899481 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:41.899450    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:41.899661 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:41.899608    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgph6" podUID="83707003-2551-42b5-8943-8243dd6daa80"
Sep 19 11:33:41.899718 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:41.899684    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:41.899812 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:41.899789    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpgkd" podUID="f96896a2-5494-4bc6-81c9-154860ee5f6b"
Sep 19 11:33:41.936024 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:41.935534    2584 generic.go:356] "Generic (PLEG): container finished" podID="58dbfd7134990f018404ee63672dd972" containerID="bb435473be0a67b4e318372f7fa360dca9386df86e2b2765fa1a9b3829a943e0" exitCode=0
Sep 19 11:33:41.936024 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:41.935961    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal" event={"ID":"58dbfd7134990f018404ee63672dd972","Type":"ContainerDied","Data":"bb435473be0a67b4e318372f7fa360dca9386df86e2b2765fa1a9b3829a943e0"}
Sep 19 11:33:41.954181 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:41.954128    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.ec2.internal" event={"ID":"49bc127c496c8981768be48e6fe5a22e","Type":"ContainerStarted","Data":"00f69d9ee2ec7b21b4a6180e60378b7912c430655b3f0413d53fe1226887330f"}
Sep 19 11:33:42.896925 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:42.896831    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:33:42.897400 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:42.896995    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2dhj" podUID="d0c272c9-c396-486e-b63b-6726cf1321d8"
Sep 19 11:33:42.985691 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:42.985654    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal" event={"ID":"58dbfd7134990f018404ee63672dd972","Type":"ContainerStarted","Data":"bd1d5d53a58db9f848c55bf3851814e1fc26ca6b1077d2975471305d89fc982a"}
Sep 19 11:33:43.001379 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:43.001313    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-209.ec2.internal" podStartSLOduration=4.001291497 podStartE2EDuration="4.001291497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 11:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:33:41.968176217 +0000 UTC m=+6.168206019" watchObservedRunningTime="2026-09-19 11:33:43.001291497 +0000 UTC m=+7.201321300"
Sep 19 11:33:43.459335 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:43.459290    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs\") pod \"network-metrics-daemon-pgph6\" (UID: \"83707003-2551-42b5-8943-8243dd6daa80\") " pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:43.459529 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:43.459418    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 11:33:43.459529 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:43.459493    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs podName:83707003-2551-42b5-8943-8243dd6daa80 nodeName:}" failed. No retries permitted until 2026-09-19 11:33:47.459472767 +0000 UTC m=+11.659502554 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs") pod "network-metrics-daemon-pgph6" (UID: "83707003-2551-42b5-8943-8243dd6daa80") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 11:33:43.560651 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:43.560605    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret\") pod \"global-pull-secret-syncer-dpgkd\" (UID: \"f96896a2-5494-4bc6-81c9-154860ee5f6b\") " pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:43.560851 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:43.560670    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7wwl\" (UniqueName: \"kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl\") pod \"network-check-target-m2dhj\" (UID: \"d0c272c9-c396-486e-b63b-6726cf1321d8\") " pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:33:43.560851 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:43.560762    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 11:33:43.560851 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:43.560792    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 11:33:43.560851 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:43.560812    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 11:33:43.560851 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:43.560824    2584 projected.go:196] Error preparing data for projected volume kube-api-access-w7wwl for pod openshift-network-diagnostics/network-check-target-m2dhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 11:33:43.560851 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:43.560844    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret podName:f96896a2-5494-4bc6-81c9-154860ee5f6b nodeName:}" failed. No retries permitted until 2026-09-19 11:33:47.5608245 +0000 UTC m=+11.760854284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret") pod "global-pull-secret-syncer-dpgkd" (UID: "f96896a2-5494-4bc6-81c9-154860ee5f6b") : object "kube-system"/"original-pull-secret" not registered
Sep 19 11:33:43.561386 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:43.560872    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl podName:d0c272c9-c396-486e-b63b-6726cf1321d8 nodeName:}" failed. No retries permitted until 2026-09-19 11:33:47.560857988 +0000 UTC m=+11.760887783 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w7wwl" (UniqueName: "kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl") pod "network-check-target-m2dhj" (UID: "d0c272c9-c396-486e-b63b-6726cf1321d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 11:33:43.898943 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:43.898888    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:43.899399 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:43.899027    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgph6" podUID="83707003-2551-42b5-8943-8243dd6daa80"
Sep 19 11:33:43.899457 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:43.899423    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:43.899536 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:43.899513    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpgkd" podUID="f96896a2-5494-4bc6-81c9-154860ee5f6b"
Sep 19 11:33:44.897566 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:44.897527    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:33:44.897776 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:44.897667    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2dhj" podUID="d0c272c9-c396-486e-b63b-6726cf1321d8"
Sep 19 11:33:45.897033 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:45.896997    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:45.897482 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:45.897171    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgph6" podUID="83707003-2551-42b5-8943-8243dd6daa80"
Sep 19 11:33:45.897634 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:45.897613    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:45.897753 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:45.897722    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpgkd" podUID="f96896a2-5494-4bc6-81c9-154860ee5f6b"
Sep 19 11:33:46.897330 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:46.897290    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:33:46.897786 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:46.897425    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2dhj" podUID="d0c272c9-c396-486e-b63b-6726cf1321d8"
Sep 19 11:33:47.494448 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:47.494410    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs\") pod \"network-metrics-daemon-pgph6\" (UID: \"83707003-2551-42b5-8943-8243dd6daa80\") " pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:47.494658 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:47.494606    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 11:33:47.494723 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:47.494690    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs podName:83707003-2551-42b5-8943-8243dd6daa80 nodeName:}" failed. No retries permitted until 2026-09-19 11:33:55.494672488 +0000 UTC m=+19.694702272 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs") pod "network-metrics-daemon-pgph6" (UID: "83707003-2551-42b5-8943-8243dd6daa80") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 11:33:47.595728 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:47.595066    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret\") pod \"global-pull-secret-syncer-dpgkd\" (UID: \"f96896a2-5494-4bc6-81c9-154860ee5f6b\") " pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:47.595728 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:47.595142    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7wwl\" (UniqueName: \"kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl\") pod \"network-check-target-m2dhj\" (UID: \"d0c272c9-c396-486e-b63b-6726cf1321d8\") " pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:33:47.595728 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:47.595277    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 11:33:47.595728 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:47.595294    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 11:33:47.595728 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:47.595306    2584 projected.go:196] Error preparing data for projected volume kube-api-access-w7wwl for pod openshift-network-diagnostics/network-check-target-m2dhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 11:33:47.595728 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:47.595354    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl podName:d0c272c9-c396-486e-b63b-6726cf1321d8 nodeName:}" failed. No retries permitted until 2026-09-19 11:33:55.595340549 +0000 UTC m=+19.795370329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w7wwl" (UniqueName: "kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl") pod "network-check-target-m2dhj" (UID: "d0c272c9-c396-486e-b63b-6726cf1321d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 11:33:47.595728 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:47.595401    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 11:33:47.595728 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:47.595428    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret podName:f96896a2-5494-4bc6-81c9-154860ee5f6b nodeName:}" failed. No retries permitted until 2026-09-19 11:33:55.595418158 +0000 UTC m=+19.795447940 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret") pod "global-pull-secret-syncer-dpgkd" (UID: "f96896a2-5494-4bc6-81c9-154860ee5f6b") : object "kube-system"/"original-pull-secret" not registered
Sep 19 11:33:47.897297 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:47.897271    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:47.897466 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:47.897366    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgph6" podUID="83707003-2551-42b5-8943-8243dd6daa80"
Sep 19 11:33:47.897750 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:47.897652    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:47.897750 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:47.897729    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpgkd" podUID="f96896a2-5494-4bc6-81c9-154860ee5f6b"
Sep 19 11:33:48.897449 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:48.896945    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:33:48.897449 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:48.897077    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2dhj" podUID="d0c272c9-c396-486e-b63b-6726cf1321d8"
Sep 19 11:33:49.897214 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:49.896701    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:49.897214 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:49.896846    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpgkd" podUID="f96896a2-5494-4bc6-81c9-154860ee5f6b"
Sep 19 11:33:49.897214 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:49.897027    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:49.897214 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:49.897121    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgph6" podUID="83707003-2551-42b5-8943-8243dd6daa80"
Sep 19 11:33:50.896979 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:50.896945    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:33:50.897159 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:50.897068    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2dhj" podUID="d0c272c9-c396-486e-b63b-6726cf1321d8"
Sep 19 11:33:51.897607 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:51.897560    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:51.898096 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:51.897613    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:51.898096 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:51.897705    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgph6" podUID="83707003-2551-42b5-8943-8243dd6daa80"
Sep 19 11:33:51.898096 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:51.897818    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpgkd" podUID="f96896a2-5494-4bc6-81c9-154860ee5f6b"
Sep 19 11:33:52.897461 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:52.897425    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:33:52.897646 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:52.897554    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2dhj" podUID="d0c272c9-c396-486e-b63b-6726cf1321d8"
Sep 19 11:33:53.897502 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:53.897463    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:53.897712 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:53.897512    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:53.897712 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:53.897623    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgph6" podUID="83707003-2551-42b5-8943-8243dd6daa80"
Sep 19 11:33:53.897712 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:53.897705    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpgkd" podUID="f96896a2-5494-4bc6-81c9-154860ee5f6b"
Sep 19 11:33:54.897582 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:54.897548    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:33:54.897804 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:54.897669    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2dhj" podUID="d0c272c9-c396-486e-b63b-6726cf1321d8"
Sep 19 11:33:55.557929 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:55.557883    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs\") pod \"network-metrics-daemon-pgph6\" (UID: \"83707003-2551-42b5-8943-8243dd6daa80\") " pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:55.558110 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:55.558067    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 11:33:55.558167 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:55.558152    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs podName:83707003-2551-42b5-8943-8243dd6daa80 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:11.558128163 +0000 UTC m=+35.758157959 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs") pod "network-metrics-daemon-pgph6" (UID: "83707003-2551-42b5-8943-8243dd6daa80") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 11:33:55.658752 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:55.658706    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7wwl\" (UniqueName: \"kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl\") pod \"network-check-target-m2dhj\" (UID: \"d0c272c9-c396-486e-b63b-6726cf1321d8\") " pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:33:55.658962 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:55.658769    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret\") pod \"global-pull-secret-syncer-dpgkd\" (UID: \"f96896a2-5494-4bc6-81c9-154860ee5f6b\") " pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:55.658962 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:55.658874    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 11:33:55.658962 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:55.658895    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 11:33:55.658962 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:55.658940    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 11:33:55.658962 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:55.658947    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret podName:f96896a2-5494-4bc6-81c9-154860ee5f6b nodeName:}" failed. No retries permitted until 2026-09-19 11:34:11.65893234 +0000 UTC m=+35.858962120 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret") pod "global-pull-secret-syncer-dpgkd" (UID: "f96896a2-5494-4bc6-81c9-154860ee5f6b") : object "kube-system"/"original-pull-secret" not registered
Sep 19 11:33:55.658962 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:55.658955    2584 projected.go:196] Error preparing data for projected volume kube-api-access-w7wwl for pod openshift-network-diagnostics/network-check-target-m2dhj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 11:33:55.659229 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:55.659007    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl podName:d0c272c9-c396-486e-b63b-6726cf1321d8 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:11.658991782 +0000 UTC m=+35.859021564 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w7wwl" (UniqueName: "kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl") pod "network-check-target-m2dhj" (UID: "d0c272c9-c396-486e-b63b-6726cf1321d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 11:33:55.897523 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:55.897480    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:55.897523 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:55.897511    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:55.897753 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:55.897630    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpgkd" podUID="f96896a2-5494-4bc6-81c9-154860ee5f6b"
Sep 19 11:33:55.897753 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:55.897711    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgph6" podUID="83707003-2551-42b5-8943-8243dd6daa80"
Sep 19 11:33:56.897625 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:56.897588    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:33:56.898067 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:56.897700    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2dhj" podUID="d0c272c9-c396-486e-b63b-6726cf1321d8"
Sep 19 11:33:57.897721 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:57.897680    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:57.898170 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:57.897729    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:57.898170 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:57.897808    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgph6" podUID="83707003-2551-42b5-8943-8243dd6daa80"
Sep 19 11:33:57.898170 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:57.897904    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpgkd" podUID="f96896a2-5494-4bc6-81c9-154860ee5f6b"
Sep 19 11:33:58.897132 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:58.896866    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:33:58.897264 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:58.897176    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2dhj" podUID="d0c272c9-c396-486e-b63b-6726cf1321d8"
Sep 19 11:33:59.019563 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.019320    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rsnfw" event={"ID":"b2e7da35-ed65-4cd7-8d44-9b25ff53fba2","Type":"ContainerStarted","Data":"44c1dc3a1ff6b6a5104c4656eb675377cb0e6a523057867588d1a6098c7f1942"}
Sep 19 11:33:59.021163 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.021133    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6ts8x" event={"ID":"b1fb8bd8-ce45-4e39-a7df-96f844fa2825","Type":"ContainerStarted","Data":"d3158feb748d17bf072e1fcfa0f98b340936f89c1f95a2496019b54688b1500a"}
Sep 19 11:33:59.024438 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.024415    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6th98_2fed27e0-833f-49a8-b11e-8f61df6e6bd5/ovn-acl-logging/0.log"
Sep 19 11:33:59.024859 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.024834    2584 generic.go:356] "Generic (PLEG): container finished" podID="2fed27e0-833f-49a8-b11e-8f61df6e6bd5" containerID="cbcfc23003966dbe1dd11265c190ea151b46fecac470d75ed7b467acdc64b207" exitCode=1
Sep 19 11:33:59.024981 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.024904    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6th98" event={"ID":"2fed27e0-833f-49a8-b11e-8f61df6e6bd5","Type":"ContainerStarted","Data":"8097fff094a416eff6991fc5354404eb43f4e5fa8feb6ed59e8eaf461c962916"}
Sep 19 11:33:59.025032 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.025004    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6th98" event={"ID":"2fed27e0-833f-49a8-b11e-8f61df6e6bd5","Type":"ContainerStarted","Data":"a3855b6ef943883e939851546a8f70b1f8fc5df0ce3467a81eddfee46813b805"}
Sep 19 11:33:59.025080 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.025052    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6th98" event={"ID":"2fed27e0-833f-49a8-b11e-8f61df6e6bd5","Type":"ContainerStarted","Data":"bfc2c7f77d98853b9fa313d359d2992b91c89912ba93fb2173dedba6d18a423e"}
Sep 19 11:33:59.025132 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.025098    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6th98" event={"ID":"2fed27e0-833f-49a8-b11e-8f61df6e6bd5","Type":"ContainerStarted","Data":"0c1dfe05fca5678a57cc18ebfe2f1333bb89305265e9946d0ab4e03d64f64327"}
Sep 19 11:33:59.025182 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.025145    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6th98" event={"ID":"2fed27e0-833f-49a8-b11e-8f61df6e6bd5","Type":"ContainerDied","Data":"cbcfc23003966dbe1dd11265c190ea151b46fecac470d75ed7b467acdc64b207"}
Sep 19 11:33:59.025233 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.025184    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6th98" event={"ID":"2fed27e0-833f-49a8-b11e-8f61df6e6bd5","Type":"ContainerStarted","Data":"c9f585a29b781acd671fbf7a1d413abceac15103d9bfd2a7232148813cedb5ce"}
Sep 19 11:33:59.026674 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.026652    2584 generic.go:356] "Generic (PLEG): container finished" podID="8dd0a400-b99d-4d02-b42c-f05d4f40a484" containerID="43b4a9b4d993a0752eeafcefa7047603b2fffaea790e2259816ff9d33eeeaa01" exitCode=0
Sep 19 11:33:59.026772 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.026735    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntx42" event={"ID":"8dd0a400-b99d-4d02-b42c-f05d4f40a484","Type":"ContainerDied","Data":"43b4a9b4d993a0752eeafcefa7047603b2fffaea790e2259816ff9d33eeeaa01"}
Sep 19 11:33:59.028213 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.028159    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vc5vt" event={"ID":"869b7a50-52b6-4167-aa2d-f5152f2c8eb4","Type":"ContainerStarted","Data":"ad8a4f08dff9f556633b5733105e097d63dda0f6f173216593881e7a85db1a95"}
Sep 19 11:33:59.029861 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.029836    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dktw9" event={"ID":"ee2c4ee6-287a-4768-9cba-cb8729cef657","Type":"ContainerStarted","Data":"a965561738301d6a8206fb308949b0962db452f68d3e2e33d0996c61aa228e87"}
Sep 19 11:33:59.031555 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.031532    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6pbcp" event={"ID":"11fdca9a-6699-4e5e-b55e-78ca9308ed2e","Type":"ContainerStarted","Data":"a3614f2fdbb1d228d005ceb6671e8bcce30875f735206bf12999df13455a2ba0"}
Sep 19 11:33:59.032573 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.032516    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rsnfw" podStartSLOduration=3.987058629 podStartE2EDuration="21.032500615s" totalImagesPullingTime="17.045441986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 11:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:33:59.03229689 +0000 UTC m=+23.232326705" watchObservedRunningTime="2026-09-19 11:33:59.032500615 +0000 UTC m=+23.232530418"
Sep 19 11:33:59.032678 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.032626    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-209.ec2.internal" podStartSLOduration=20.032616741 podStartE2EDuration="20.032616741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 11:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:33:43.001570255 +0000 UTC m=+7.201600057" watchObservedRunningTime="2026-09-19 11:33:59.032616741 +0000 UTC m=+23.232646570"
Sep 19 11:33:59.033802 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.033782    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml" event={"ID":"2b566375-2b61-4d12-b35d-6c019ef72fb1","Type":"ContainerStarted","Data":"54e3561f00f32e37767e1c3f5d4bacae5ef0d01cc07643fc50b6196457842666"}
Sep 19 11:33:59.063928 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.063869    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vc5vt" podStartSLOduration=3.993687001 podStartE2EDuration="21.06385288s" totalImagesPullingTime="17.070165879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 11:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:33:59.045477121 +0000 UTC m=+23.245506923" watchObservedRunningTime="2026-09-19 11:33:59.06385288 +0000 UTC m=+23.263882682"
Sep 19 11:33:59.064262 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.064241    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6pbcp" podStartSLOduration=3.6492297430000002 podStartE2EDuration="21.064236037s" totalImagesPullingTime="17.415006294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 11:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:33:59.063695407 +0000 UTC m=+23.263725210" watchObservedRunningTime="2026-09-19 11:33:59.064236037 +0000 UTC m=+23.264265839"
Sep 19 11:33:59.095702 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.095645    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dktw9" podStartSLOduration=7.705018933 podStartE2EDuration="21.095628713s" totalImagesPullingTime="13.39060978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:33:59.095491183 +0000 UTC m=+23.295520985" watchObservedRunningTime="2026-09-19 11:33:59.095628713 +0000 UTC m=+23.295658514"
Sep 19 11:33:59.113487 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.113432    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6ts8x" podStartSLOduration=4.034795884 podStartE2EDuration="21.113414061s" totalImagesPullingTime="17.078618177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 11:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:33:59.113002119 +0000 UTC m=+23.313031921" watchObservedRunningTime="2026-09-19 11:33:59.113414061 +0000 UTC m=+23.313443878"
Sep 19 11:33:59.831443 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.831414    2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 19 11:33:59.897647 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.897552    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:33:59.897647 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.897605    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:33:59.897871 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:59.897702    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgph6" podUID="83707003-2551-42b5-8943-8243dd6daa80"
Sep 19 11:33:59.897871 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:33:59.897819    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpgkd" podUID="f96896a2-5494-4bc6-81c9-154860ee5f6b"
Sep 19 11:33:59.902800 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.902707    2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-19T11:33:59.831436249Z","UUID":"af4085f5-d688-4a76-8b25-cc80754f18ac","Handler":null,"Name":"","Endpoint":""}
Sep 19 11:33:59.904374 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.904354    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 19 11:33:59.904374 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:33:59.904387    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 19 11:34:00.038275 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:00.038234    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml" event={"ID":"2b566375-2b61-4d12-b35d-6c019ef72fb1","Type":"ContainerStarted","Data":"d128131ea8ec05ea76eb191f367d9759622cc0cefb0ce263561865ad1a8c5e75"}
Sep 19 11:34:00.039970 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:00.039934    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ndsnt" event={"ID":"e84ed163-05a3-45ce-95b9-b5236328e3e3","Type":"ContainerStarted","Data":"732a6832cdb2bd6cd708a8297685b289df5a0609d70b3d3988e9635fdc2dd0d1"}
Sep 19 11:34:00.897428 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:00.897394    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:34:00.897563 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:00.897528    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2dhj" podUID="d0c272c9-c396-486e-b63b-6726cf1321d8"
Sep 19 11:34:01.047051 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:01.047026    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6th98_2fed27e0-833f-49a8-b11e-8f61df6e6bd5/ovn-acl-logging/0.log"
Sep 19 11:34:01.047517 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:01.047492    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6th98" event={"ID":"2fed27e0-833f-49a8-b11e-8f61df6e6bd5","Type":"ContainerStarted","Data":"3ab6f87259c1d4de6fa28079e60cde0548e50ad2e69c5fc63e11eb5b05972454"}
Sep 19 11:34:01.050525 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:01.050493    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml" event={"ID":"2b566375-2b61-4d12-b35d-6c019ef72fb1","Type":"ContainerStarted","Data":"62f19a4f4927d12fb7e6295c186beb357dcae685d66b0723a96d891bf50be585"}
Sep 19 11:34:01.068567 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:01.068515    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ndsnt" podStartSLOduration=5.981561937 podStartE2EDuration="23.06849732s" totalImagesPullingTime="17.086935383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 11:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:34:00.053784011 +0000 UTC m=+24.253813816" watchObservedRunningTime="2026-09-19 11:34:01.06849732 +0000 UTC m=+25.268527125"
Sep 19 11:34:01.069018 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:01.068985    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d9ml" podStartSLOduration=3.474231658 podStartE2EDuration="23.068977156s" totalImagesPullingTime="19.594745498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 11:33:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:34:01.068460779 +0000 UTC m=+25.268490582" watchObservedRunningTime="2026-09-19 11:34:01.068977156 +0000 UTC m=+25.269006958"
Sep 19 11:34:01.897704 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:01.897663    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:34:01.897868 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:01.897711    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:34:01.897868 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:01.897822    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgph6" podUID="83707003-2551-42b5-8943-8243dd6daa80"
Sep 19 11:34:01.897983 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:01.897926    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpgkd" podUID="f96896a2-5494-4bc6-81c9-154860ee5f6b"
Sep 19 11:34:02.897646 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:02.897605    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:34:02.898130 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:02.897734    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2dhj" podUID="d0c272c9-c396-486e-b63b-6726cf1321d8"
Sep 19 11:34:03.813618 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:03.813445    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dktw9"
Sep 19 11:34:03.814070 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:03.814048    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dktw9"
Sep 19 11:34:03.897306 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:03.897277    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:34:03.897471 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:03.897276    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:34:03.897471 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:03.897426    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgph6" podUID="83707003-2551-42b5-8943-8243dd6daa80"
Sep 19 11:34:03.897602 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:03.897479    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpgkd" podUID="f96896a2-5494-4bc6-81c9-154860ee5f6b"
Sep 19 11:34:04.061189 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:04.061162    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6th98_2fed27e0-833f-49a8-b11e-8f61df6e6bd5/ovn-acl-logging/0.log"
Sep 19 11:34:04.062045 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:04.061616    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6th98" event={"ID":"2fed27e0-833f-49a8-b11e-8f61df6e6bd5","Type":"ContainerStarted","Data":"d83ceb8fb50a13d8dd1ee90ce051dd24428a855d773949426f4857e2c0b83964"}
Sep 19 11:34:04.062045 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:04.061891    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:34:04.062217 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:04.062200    2584 scope.go:117] "RemoveContainer" containerID="cbcfc23003966dbe1dd11265c190ea151b46fecac470d75ed7b467acdc64b207"
Sep 19 11:34:04.064389 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:04.064362    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntx42" event={"ID":"8dd0a400-b99d-4d02-b42c-f05d4f40a484","Type":"ContainerStarted","Data":"d3cc112a7ec994ae550abe0e57a9f834f188e04aa1094d073104fc8dcc010fed"}
Sep 19 11:34:04.064766 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:04.064725    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dktw9"
Sep 19 11:34:04.065002 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:04.064981    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dktw9"
Sep 19 11:34:04.079188 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:04.079162    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:34:04.896710 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:04.896677    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:34:04.896879 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:04.896789    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2dhj" podUID="d0c272c9-c396-486e-b63b-6726cf1321d8"
Sep 19 11:34:05.071588 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:05.071565    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6th98_2fed27e0-833f-49a8-b11e-8f61df6e6bd5/ovn-acl-logging/0.log"
Sep 19 11:34:05.072070 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:05.072046    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6th98" event={"ID":"2fed27e0-833f-49a8-b11e-8f61df6e6bd5","Type":"ContainerStarted","Data":"1d3aab8a3a605cabc762ee4ec4e8be2ad6d8d41d4cb1f2e85b8d898070a186c2"}
Sep 19 11:34:05.072162 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:05.072127    2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 19 11:34:05.072441 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:05.072424    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:34:05.075111 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:05.075081    2584 generic.go:356] "Generic (PLEG): container finished" podID="8dd0a400-b99d-4d02-b42c-f05d4f40a484" containerID="d3cc112a7ec994ae550abe0e57a9f834f188e04aa1094d073104fc8dcc010fed" exitCode=0
Sep 19 11:34:05.075290 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:05.075169    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntx42" event={"ID":"8dd0a400-b99d-4d02-b42c-f05d4f40a484","Type":"ContainerDied","Data":"d3cc112a7ec994ae550abe0e57a9f834f188e04aa1094d073104fc8dcc010fed"}
Sep 19 11:34:05.101247 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:05.101191    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6th98" podStartSLOduration=10.001867999 podStartE2EDuration="27.101170687s" totalImagesPullingTime="17.099302688s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 11:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:34:05.100792755 +0000 UTC m=+29.300822559" watchObservedRunningTime="2026-09-19 11:34:05.101170687 +0000 UTC m=+29.301200491"
Sep 19 11:34:05.105303 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:05.105264    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:34:05.694222 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:05.694005    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dpgkd"]
Sep 19 11:34:05.694427 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:05.694341    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:34:05.694494 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:05.694471    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpgkd" podUID="f96896a2-5494-4bc6-81c9-154860ee5f6b"
Sep 19 11:34:05.698896 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:05.697459    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m2dhj"]
Sep 19 11:34:05.698896 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:05.697526    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pgph6"]
Sep 19 11:34:05.698896 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:05.697616    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:34:05.698896 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:05.697726    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgph6" podUID="83707003-2551-42b5-8943-8243dd6daa80"
Sep 19 11:34:05.698896 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:05.697783    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:34:05.698896 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:05.697866    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2dhj" podUID="d0c272c9-c396-486e-b63b-6726cf1321d8"
Sep 19 11:34:06.077414 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:06.077379    2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 19 11:34:06.897463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:06.897427    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:34:06.897646 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:06.897535    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2dhj" podUID="d0c272c9-c396-486e-b63b-6726cf1321d8"
Sep 19 11:34:06.897646 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:06.897587    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:34:06.897715 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:06.897679    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpgkd" podUID="f96896a2-5494-4bc6-81c9-154860ee5f6b"
Sep 19 11:34:07.081956 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:07.081895    2584 generic.go:356] "Generic (PLEG): container finished" podID="8dd0a400-b99d-4d02-b42c-f05d4f40a484" containerID="448cb2fbb744c9d1c358d06052e8695850eb7bafa3e93d84619e8644a5518c9c" exitCode=0
Sep 19 11:34:07.081956 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:07.081940    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntx42" event={"ID":"8dd0a400-b99d-4d02-b42c-f05d4f40a484","Type":"ContainerDied","Data":"448cb2fbb744c9d1c358d06052e8695850eb7bafa3e93d84619e8644a5518c9c"}
Sep 19 11:34:07.082338 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:07.082191    2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 19 11:34:07.897420 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:07.897391    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:34:07.897607 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:07.897504    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgph6" podUID="83707003-2551-42b5-8943-8243dd6daa80"
Sep 19 11:34:08.087393 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:08.087358    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntx42" event={"ID":"8dd0a400-b99d-4d02-b42c-f05d4f40a484","Type":"ContainerStarted","Data":"d3f427c33a512001597371e7171013f5608b44f5dc08be9d02b79cdab1ae7338"}
Sep 19 11:34:08.897376 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:08.897334    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:34:08.897574 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:08.897463    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dpgkd" podUID="f96896a2-5494-4bc6-81c9-154860ee5f6b"
Sep 19 11:34:08.897574 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:08.897521    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:34:08.897686 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:08.897628    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2dhj" podUID="d0c272c9-c396-486e-b63b-6726cf1321d8"
Sep 19 11:34:09.091987 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:09.091955    2584 generic.go:356] "Generic (PLEG): container finished" podID="8dd0a400-b99d-4d02-b42c-f05d4f40a484" containerID="d3f427c33a512001597371e7171013f5608b44f5dc08be9d02b79cdab1ae7338" exitCode=0
Sep 19 11:34:09.092361 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:09.092011    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntx42" event={"ID":"8dd0a400-b99d-4d02-b42c-f05d4f40a484","Type":"ContainerDied","Data":"d3f427c33a512001597371e7171013f5608b44f5dc08be9d02b79cdab1ae7338"}
Sep 19 11:34:09.493250 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:09.493216    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:34:09.493464 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:09.493448    2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 19 11:34:09.512072 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:09.512044    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6th98"
Sep 19 11:34:09.897553 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:09.897524    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:34:09.897714 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:09.897667    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgph6" podUID="83707003-2551-42b5-8943-8243dd6daa80"
Sep 19 11:34:10.862904 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.862874    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-209.ec2.internal" event="NodeReady"
Sep 19 11:34:10.863315 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.863049    2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 19 11:34:10.896502 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.896460    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-m9jqg"]
Sep 19 11:34:10.899192 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.899144    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:34:10.899192 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.899194    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-m9jqg"
Sep 19 11:34:10.899576 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.899545    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:34:10.900001 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.899976    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-4bxfj"]
Sep 19 11:34:10.901445 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.901425    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-687866bb-9xvr5"]
Sep 19 11:34:10.901563 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.901546    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj"
Sep 19 11:34:10.902285 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.902262    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 19 11:34:10.902481 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.902473    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 19 11:34:10.902638 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.902620    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 19 11:34:10.903034 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.902896    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 19 11:34:10.903034 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.902905    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lhwvf\""
Sep 19 11:34:10.903034 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.902928    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 11:34:10.903034 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.902971    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"]
Sep 19 11:34:10.903269 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.903140    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:10.903524 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.903500    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 19 11:34:10.903625 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.903572    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 19 11:34:10.903625 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.903584    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-hb2vp\""
Sep 19 11:34:10.904073 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.904058    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 19 11:34:10.904158 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.904127    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 19 11:34:10.904203 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.904181    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 19 11:34:10.904344 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.904332    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7xckw\""
Sep 19 11:34:10.904518 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.904503    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 19 11:34:10.904634 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.904616    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-657b49657d-km2v8"]
Sep 19 11:34:10.904696 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.904678    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"
Sep 19 11:34:10.905961 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.905943    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 19 11:34:10.906140 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.906118    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9zcpw"]
Sep 19 11:34:10.906247 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.906157    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 19 11:34:10.906247 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.906173    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 19 11:34:10.906330 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.906245    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:10.906619 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.906548    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 19 11:34:10.907310 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.907295    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4gb6j\""
Sep 19 11:34:10.907400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.907315    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 19 11:34:10.907671 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.907645    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 19 11:34:10.907671 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.907652    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ftf42"]
Sep 19 11:34:10.907768 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.907712    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9zcpw"
Sep 19 11:34:10.908037 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.908022    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 19 11:34:10.909061 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.909046    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 19 11:34:10.909232 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.909217    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ftf42"
Sep 19 11:34:10.910806 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.910787    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-j2npf\""
Sep 19 11:34:10.910896 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.910805    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 19 11:34:10.910896 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.910813    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 19 11:34:10.911450 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.911430    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 19 11:34:10.911536 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.911464    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 19 11:34:10.911755 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.911739    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 19 11:34:10.912042 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.912028    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 19 11:34:10.912178 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.912157    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 19 11:34:10.912262 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.912222    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sbk8r\""
Sep 19 11:34:10.912326 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.912300    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xbbzs\""
Sep 19 11:34:10.914823 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.914807    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 19 11:34:10.914939 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.914828    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qs7cc\""
Sep 19 11:34:10.915110 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.915052    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 19 11:34:10.915570 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.915554    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 19 11:34:10.915649 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.915581    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 19 11:34:10.916418 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.916400    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 19 11:34:10.918854 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.918836    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 19 11:34:10.921594 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.920213    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-m9jqg"]
Sep 19 11:34:10.921756 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.921736    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-t27gs"]
Sep 19 11:34:10.937425 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.937402    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"]
Sep 19 11:34:10.937611 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.937431    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-4bxfj"]
Sep 19 11:34:10.937611 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.937451    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk"]
Sep 19 11:34:10.937611 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.937533    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:10.941419 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.941288    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-7wgwd"]
Sep 19 11:34:10.941926 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.941889    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 19 11:34:10.943143 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.942135    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 19 11:34:10.943710 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.943691    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mvj2p\""
Sep 19 11:34:10.944624 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.944602    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk"
Sep 19 11:34:10.947382 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.947363    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 19 11:34:10.948382 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.947373    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 19 11:34:10.949182 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.947388    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 19 11:34:10.949397 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.947448    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5jm8q\""
Sep 19 11:34:10.949636 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.949617    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h4xjv"]
Sep 19 11:34:10.949751 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.949735    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:10.953768 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.953750    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-hlvdj"]
Sep 19 11:34:10.954490 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.954472    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 19 11:34:10.954879 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.954859    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-tkhg7\""
Sep 19 11:34:10.955081 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.955064    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 19 11:34:10.955302 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.955284    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 19 11:34:10.955527 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.955510    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 19 11:34:10.955906 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.955882    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h4xjv"
Sep 19 11:34:10.960273 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.960252    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5774c959c7-6f759"]
Sep 19 11:34:10.960367 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.960278    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-t9jnr\""
Sep 19 11:34:10.960367 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.960344    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 19 11:34:10.960524 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.960508    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 19 11:34:10.960637 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.960620    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hlvdj"
Sep 19 11:34:10.960713 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.960696    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 19 11:34:10.963927 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.963290    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6mgch\""
Sep 19 11:34:10.964037 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.963981    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5774c959c7-6f759"
Sep 19 11:34:10.964878 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.964852    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-r6svd"]
Sep 19 11:34:10.965687 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.965632    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 19 11:34:10.966515 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.966496    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 19 11:34:10.966603 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.966549    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 19 11:34:10.966603 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.966589    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 19 11:34:10.966714 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.966678    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 19 11:34:10.968618 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.968601    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"]
Sep 19 11:34:10.968892 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.968876    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-r6svd"
Sep 19 11:34:10.970723 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.970697    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-stats-auth\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:10.970884 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.970865    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:10.971025 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.970998    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:10.971104 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.971007    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-certificates\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:10.971157 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.971105    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa857849-d4f4-4815-aa2d-8412e4a8033a-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-ftf42\" (UID: \"aa857849-d4f4-4815-aa2d-8412e4a8033a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ftf42"
Sep 19 11:34:10.971157 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.971144    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nll25\" (UID: \"53ad775e-63df-4c3d-856d-f17723aed41e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"
Sep 19 11:34:10.971255 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.971176    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:10.971255 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.971205    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96c0a054-f70d-46dd-b510-9056a47a6b9b-serving-cert\") pod \"service-ca-operator-57cdc4b768-m9jqg\" (UID: \"96c0a054-f70d-46dd-b510-9056a47a6b9b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-m9jqg"
Sep 19 11:34:10.971255 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.971235    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpm6l\" (UniqueName: \"kubernetes.io/projected/40987fee-2f7a-4681-a32c-22aaa7b8a8f7-kube-api-access-hpm6l\") pod \"volume-data-source-validator-5c4db8bdf9-9zcpw\" (UID: \"40987fee-2f7a-4681-a32c-22aaa7b8a8f7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9zcpw"
Sep 19 11:34:10.971380 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.971296    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60111092-bf1d-4949-b3a2-fc36e4a01d59-image-registry-private-configuration\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:10.971380 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.971334    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60111092-bf1d-4949-b3a2-fc36e4a01d59-trusted-ca\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:10.971380 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.971361    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjmxg\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-kube-api-access-bjmxg\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:10.971517 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.971419    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24aae5c7-e112-496e-ba52-a52e02e520c1-config\") pod \"console-operator-fd4bb46c-4bxfj\" (UID: \"24aae5c7-e112-496e-ba52-a52e02e520c1\") " pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj"
Sep 19 11:34:10.971517 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.971452    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60111092-bf1d-4949-b3a2-fc36e4a01d59-ca-trust-extracted\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:10.971517 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.971488    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9h65\" (UniqueName: \"kubernetes.io/projected/24aae5c7-e112-496e-ba52-a52e02e520c1-kube-api-access-f9h65\") pod \"console-operator-fd4bb46c-4bxfj\" (UID: \"24aae5c7-e112-496e-ba52-a52e02e520c1\") " pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj"
Sep 19 11:34:10.971663 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.971517    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 19 11:34:10.971735 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.971517    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa857849-d4f4-4815-aa2d-8412e4a8033a-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-ftf42\" (UID: \"aa857849-d4f4-4815-aa2d-8412e4a8033a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ftf42"
Sep 19 11:34:10.971827 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.971794    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-bound-sa-token\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:10.971931 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.971889    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78jrp\" (UniqueName: \"kubernetes.io/projected/5b388e80-43fb-4367-a7c1-6c53f04caffa-kube-api-access-78jrp\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:10.971993 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.971948    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96c0a054-f70d-46dd-b510-9056a47a6b9b-config\") pod \"service-ca-operator-57cdc4b768-m9jqg\" (UID: \"96c0a054-f70d-46dd-b510-9056a47a6b9b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-m9jqg"
Sep 19 11:34:10.971993 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.971984    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/53ad775e-63df-4c3d-856d-f17723aed41e-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-nll25\" (UID: \"53ad775e-63df-4c3d-856d-f17723aed41e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"
Sep 19 11:34:10.972110 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.972014    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:10.972110 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.972044    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60111092-bf1d-4949-b3a2-fc36e4a01d59-installation-pull-secrets\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:10.972110 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.972071    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2psp\" (UniqueName: \"kubernetes.io/projected/aa857849-d4f4-4815-aa2d-8412e4a8033a-kube-api-access-s2psp\") pod \"kube-storage-version-migrator-operator-55889f7b47-ftf42\" (UID: \"aa857849-d4f4-4815-aa2d-8412e4a8033a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ftf42"
Sep 19 11:34:10.972110 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.972104    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24aae5c7-e112-496e-ba52-a52e02e520c1-serving-cert\") pod \"console-operator-fd4bb46c-4bxfj\" (UID: \"24aae5c7-e112-496e-ba52-a52e02e520c1\") " pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj"
Sep 19 11:34:10.972287 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.972127    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24aae5c7-e112-496e-ba52-a52e02e520c1-trusted-ca\") pod \"console-operator-fd4bb46c-4bxfj\" (UID: \"24aae5c7-e112-496e-ba52-a52e02e520c1\") " pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj"
Sep 19 11:34:10.972287 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.972177    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4tsq\" (UniqueName: \"kubernetes.io/projected/53ad775e-63df-4c3d-856d-f17723aed41e-kube-api-access-t4tsq\") pod \"cluster-monitoring-operator-847b8bd8d7-nll25\" (UID: \"53ad775e-63df-4c3d-856d-f17723aed41e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"
Sep 19 11:34:10.972287 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.972205    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-default-certificate\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:10.972287 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.972234    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spblr\" (UniqueName: \"kubernetes.io/projected/96c0a054-f70d-46dd-b510-9056a47a6b9b-kube-api-access-spblr\") pod \"service-ca-operator-57cdc4b768-m9jqg\" (UID: \"96c0a054-f70d-46dd-b510-9056a47a6b9b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-m9jqg"
Sep 19 11:34:10.972573 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.972556    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xhcrd\""
Sep 19 11:34:10.973089 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.973069    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 19 11:34:10.973236 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.973215    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm"]
Sep 19 11:34:10.974085 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.974063    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:10.976945 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.976905    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 19 11:34:10.977234 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.977204    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 19 11:34:10.977452 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.977429    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 19 11:34:10.977546 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.977217    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 19 11:34:10.979102 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.979056    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-687866bb-9xvr5"]
Sep 19 11:34:10.979325 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.979306    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-657b49657d-km2v8"]
Sep 19 11:34:10.979433 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.979334    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h4xjv"]
Sep 19 11:34:10.979433 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.979356    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9zcpw"]
Sep 19 11:34:10.979433 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.979378    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-7wgwd"]
Sep 19 11:34:10.979433 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.979389    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk"]
Sep 19 11:34:10.979433 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.979408    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ftf42"]
Sep 19 11:34:10.979433 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.979423    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5774c959c7-6f759"]
Sep 19 11:34:10.979433 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.979435    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-r6svd"]
Sep 19 11:34:10.979433 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.979447    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t27gs"]
Sep 19 11:34:10.979433 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.979462    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-hlvdj"]
Sep 19 11:34:10.979981 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.979473    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm"]
Sep 19 11:34:10.979981 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.979497    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"]
Sep 19 11:34:10.979981 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.979193    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm"
Sep 19 11:34:10.982374 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.982355    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 19 11:34:10.982716 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:10.982695    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-bz95m\""
Sep 19 11:34:11.061877 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.061854    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:11.073114 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073091    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nll25\" (UID: \"53ad775e-63df-4c3d-856d-f17723aed41e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"
Sep 19 11:34:11.073215 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073118    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96c0a054-f70d-46dd-b510-9056a47a6b9b-serving-cert\") pod \"service-ca-operator-57cdc4b768-m9jqg\" (UID: \"96c0a054-f70d-46dd-b510-9056a47a6b9b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-m9jqg"
Sep 19 11:34:11.073215 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073137    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gst2x\" (UniqueName: \"kubernetes.io/projected/c323067d-ee96-4a29-8322-83ce5f3f454f-kube-api-access-gst2x\") pod \"cluster-proxy-proxy-agent-6cc5f998dc-9cvsg\" (UID: \"c323067d-ee96-4a29-8322-83ce5f3f454f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:11.073215 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073157    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sjwt\" (UniqueName: \"kubernetes.io/projected/36391cd8-d035-4be4-bcb3-f33a076dd18a-kube-api-access-2sjwt\") pod \"managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm\" (UID: \"36391cd8-d035-4be4-bcb3-f33a076dd18a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm"
Sep 19 11:34:11.073215 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073176    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jdq2\" (UniqueName: \"kubernetes.io/projected/cc7e845c-b6fd-4a96-8258-80b4e8f02e57-kube-api-access-7jdq2\") pod \"insights-operator-6f9c644dd9-7wgwd\" (UID: \"cc7e845c-b6fd-4a96-8258-80b4e8f02e57\") " pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:11.073363 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.073240    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 11:34:11.073363 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073248    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgmzc\" (UniqueName: \"kubernetes.io/projected/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-kube-api-access-qgmzc\") pod \"dns-default-t27gs\" (UID: \"7b681f4b-5ee2-4693-8e41-4be4f0a374c1\") " pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:11.073363 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.073322    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls podName:53ad775e-63df-4c3d-856d-f17723aed41e nodeName:}" failed. No retries permitted until 2026-09-19 11:34:11.57330254 +0000 UTC m=+35.773332334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-nll25" (UID: "53ad775e-63df-4c3d-856d-f17723aed41e") : secret "cluster-monitoring-operator-tls" not found
Sep 19 11:34:11.073363 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073341    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60111092-bf1d-4949-b3a2-fc36e4a01d59-image-registry-private-configuration\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:11.073493 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073364    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24aae5c7-e112-496e-ba52-a52e02e520c1-config\") pod \"console-operator-fd4bb46c-4bxfj\" (UID: \"24aae5c7-e112-496e-ba52-a52e02e520c1\") " pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj"
Sep 19 11:34:11.073493 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073391    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c323067d-ee96-4a29-8322-83ce5f3f454f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cc5f998dc-9cvsg\" (UID: \"c323067d-ee96-4a29-8322-83ce5f3f454f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:11.073493 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073418    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9h65\" (UniqueName: \"kubernetes.io/projected/24aae5c7-e112-496e-ba52-a52e02e520c1-kube-api-access-f9h65\") pod \"console-operator-fd4bb46c-4bxfj\" (UID: \"24aae5c7-e112-496e-ba52-a52e02e520c1\") " pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj"
Sep 19 11:34:11.073493 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073440    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c323067d-ee96-4a29-8322-83ce5f3f454f-hub\") pod \"cluster-proxy-proxy-agent-6cc5f998dc-9cvsg\" (UID: \"c323067d-ee96-4a29-8322-83ce5f3f454f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:11.073493 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073463    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36391cd8-d035-4be4-bcb3-f33a076dd18a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm\" (UID: \"36391cd8-d035-4be4-bcb3-f33a076dd18a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm"
Sep 19 11:34:11.073830 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073494    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-r6svd\" (UID: \"83c3531f-3ddd-45b9-91cd-3ac141ebd487\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-r6svd"
Sep 19 11:34:11.073830 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073616    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spblr\" (UniqueName: \"kubernetes.io/projected/96c0a054-f70d-46dd-b510-9056a47a6b9b-kube-api-access-spblr\") pod \"service-ca-operator-57cdc4b768-m9jqg\" (UID: \"96c0a054-f70d-46dd-b510-9056a47a6b9b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-m9jqg"
Sep 19 11:34:11.073830 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073657    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96c0a054-f70d-46dd-b510-9056a47a6b9b-config\") pod \"service-ca-operator-57cdc4b768-m9jqg\" (UID: \"96c0a054-f70d-46dd-b510-9056a47a6b9b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-m9jqg"
Sep 19 11:34:11.073830 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073692    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c323067d-ee96-4a29-8322-83ce5f3f454f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cc5f998dc-9cvsg\" (UID: \"c323067d-ee96-4a29-8322-83ce5f3f454f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:11.073830 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073717    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2066d78e-223c-4531-98aa-edc826b41c3b-cert\") pod \"ingress-canary-h4xjv\" (UID: \"2066d78e-223c-4531-98aa-edc826b41c3b\") " pod="openshift-ingress-canary/ingress-canary-h4xjv"
Sep 19 11:34:11.073830 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073757    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/53ad775e-63df-4c3d-856d-f17723aed41e-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-nll25\" (UID: \"53ad775e-63df-4c3d-856d-f17723aed41e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"
Sep 19 11:34:11.074141 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073831    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2psp\" (UniqueName: \"kubernetes.io/projected/aa857849-d4f4-4815-aa2d-8412e4a8033a-kube-api-access-s2psp\") pod \"kube-storage-version-migrator-operator-55889f7b47-ftf42\" (UID: \"aa857849-d4f4-4815-aa2d-8412e4a8033a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ftf42"
Sep 19 11:34:11.074141 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073940    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4vtk\" (UniqueName: \"kubernetes.io/projected/e153e212-b7ce-4fea-89bd-970b816f0763-kube-api-access-v4vtk\") pod \"cluster-samples-operator-685d7d68b5-q9gbk\" (UID: \"e153e212-b7ce-4fea-89bd-970b816f0763\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk"
Sep 19 11:34:11.074141 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.073980    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24aae5c7-e112-496e-ba52-a52e02e520c1-serving-cert\") pod \"console-operator-fd4bb46c-4bxfj\" (UID: \"24aae5c7-e112-496e-ba52-a52e02e520c1\") " pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj"
Sep 19 11:34:11.074141 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074023    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/20620a5d-5b06-46ca-bff8-5322e24c32b2-klusterlet-config\") pod \"klusterlet-addon-workmgr-5774c959c7-6f759\" (UID: \"20620a5d-5b06-46ca-bff8-5322e24c32b2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5774c959c7-6f759"
Sep 19 11:34:11.074141 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-certificates\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:11.074141 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074079    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24aae5c7-e112-496e-ba52-a52e02e520c1-config\") pod \"console-operator-fd4bb46c-4bxfj\" (UID: \"24aae5c7-e112-496e-ba52-a52e02e520c1\") " pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj"
Sep 19 11:34:11.074141 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074083    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpm6l\" (UniqueName: \"kubernetes.io/projected/40987fee-2f7a-4681-a32c-22aaa7b8a8f7-kube-api-access-hpm6l\") pod \"volume-data-source-validator-5c4db8bdf9-9zcpw\" (UID: \"40987fee-2f7a-4681-a32c-22aaa7b8a8f7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9zcpw"
Sep 19 11:34:11.074141 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074141    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20620a5d-5b06-46ca-bff8-5322e24c32b2-tmp\") pod \"klusterlet-addon-workmgr-5774c959c7-6f759\" (UID: \"20620a5d-5b06-46ca-bff8-5322e24c32b2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5774c959c7-6f759"
Sep 19 11:34:11.074539 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074174    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph8rl\" (UniqueName: \"kubernetes.io/projected/20620a5d-5b06-46ca-bff8-5322e24c32b2-kube-api-access-ph8rl\") pod \"klusterlet-addon-workmgr-5774c959c7-6f759\" (UID: \"20620a5d-5b06-46ca-bff8-5322e24c32b2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5774c959c7-6f759"
Sep 19 11:34:11.074539 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074200    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:11.074539 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074225    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc7e845c-b6fd-4a96-8258-80b4e8f02e57-tmp\") pod \"insights-operator-6f9c644dd9-7wgwd\" (UID: \"cc7e845c-b6fd-4a96-8258-80b4e8f02e57\") " pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:11.074539 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074249    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/83c3531f-3ddd-45b9-91cd-3ac141ebd487-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-r6svd\" (UID: \"83c3531f-3ddd-45b9-91cd-3ac141ebd487\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-r6svd"
Sep 19 11:34:11.074539 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074258    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96c0a054-f70d-46dd-b510-9056a47a6b9b-config\") pod \"service-ca-operator-57cdc4b768-m9jqg\" (UID: \"96c0a054-f70d-46dd-b510-9056a47a6b9b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-m9jqg"
Sep 19 11:34:11.074539 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074272    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls\") pod \"dns-default-t27gs\" (UID: \"7b681f4b-5ee2-4693-8e41-4be4f0a374c1\") " pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:11.074539 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074313    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60111092-bf1d-4949-b3a2-fc36e4a01d59-trusted-ca\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:11.074539 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074338    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjmxg\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-kube-api-access-bjmxg\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:11.074539 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.074370    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle podName:5b388e80-43fb-4367-a7c1-6c53f04caffa nodeName:}" failed. No retries permitted until 2026-09-19 11:34:11.574354156 +0000 UTC m=+35.774383953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle") pod "router-default-687866bb-9xvr5" (UID: "5b388e80-43fb-4367-a7c1-6c53f04caffa") : configmap references non-existent config key: service-ca.crt
Sep 19 11:34:11.074539 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074400    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc7e845c-b6fd-4a96-8258-80b4e8f02e57-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-7wgwd\" (UID: \"cc7e845c-b6fd-4a96-8258-80b4e8f02e57\") " pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:11.074539 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074431    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-q9gbk\" (UID: \"e153e212-b7ce-4fea-89bd-970b816f0763\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk"
Sep 19 11:34:11.074539 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074478    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60111092-bf1d-4949-b3a2-fc36e4a01d59-ca-trust-extracted\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:11.074539 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074541    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa857849-d4f4-4815-aa2d-8412e4a8033a-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-ftf42\" (UID: \"aa857849-d4f4-4815-aa2d-8412e4a8033a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ftf42"
Sep 19 11:34:11.075154 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074550    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/53ad775e-63df-4c3d-856d-f17723aed41e-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-nll25\" (UID: \"53ad775e-63df-4c3d-856d-f17723aed41e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"
Sep 19 11:34:11.075154 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074565    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc7e845c-b6fd-4a96-8258-80b4e8f02e57-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-7wgwd\" (UID: \"cc7e845c-b6fd-4a96-8258-80b4e8f02e57\") " pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:11.075154 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074610    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-bound-sa-token\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:11.075154 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074661    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c323067d-ee96-4a29-8322-83ce5f3f454f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cc5f998dc-9cvsg\" (UID: \"c323067d-ee96-4a29-8322-83ce5f3f454f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:11.075154 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074695    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc7e845c-b6fd-4a96-8258-80b4e8f02e57-serving-cert\") pod \"insights-operator-6f9c644dd9-7wgwd\" (UID: \"cc7e845c-b6fd-4a96-8258-80b4e8f02e57\") " pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:11.075154 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074728    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78jrp\" (UniqueName: \"kubernetes.io/projected/5b388e80-43fb-4367-a7c1-6c53f04caffa-kube-api-access-78jrp\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:11.075154 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074774    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-config-volume\") pod \"dns-default-t27gs\" (UID: \"7b681f4b-5ee2-4693-8e41-4be4f0a374c1\") " pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:11.075154 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074778    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-certificates\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:11.075154 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.074797    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c323067d-ee96-4a29-8322-83ce5f3f454f-ca\") pod \"cluster-proxy-proxy-agent-6cc5f998dc-9cvsg\" (UID: \"c323067d-ee96-4a29-8322-83ce5f3f454f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:11.075154 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.075143    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60111092-bf1d-4949-b3a2-fc36e4a01d59-ca-trust-extracted\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:11.075656 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.075170    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrslg\" (UniqueName: \"kubernetes.io/projected/1936edc6-ecd0-46ca-9b8e-08cc8b2dc383-kube-api-access-hrslg\") pod \"network-check-source-7cbc45fddc-hlvdj\" (UID: \"1936edc6-ecd0-46ca-9b8e-08cc8b2dc383\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hlvdj"
Sep 19 11:34:11.075656 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.075206    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:11.075656 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.075233    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60111092-bf1d-4949-b3a2-fc36e4a01d59-installation-pull-secrets\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:11.075656 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.075262    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24aae5c7-e112-496e-ba52-a52e02e520c1-trusted-ca\") pod \"console-operator-fd4bb46c-4bxfj\" (UID: \"24aae5c7-e112-496e-ba52-a52e02e520c1\") " pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj"
Sep 19 11:34:11.075656 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.075330    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 11:34:11.075656 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.075389    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs podName:5b388e80-43fb-4367-a7c1-6c53f04caffa nodeName:}" failed. No retries permitted until 2026-09-19 11:34:11.575374 +0000 UTC m=+35.775403813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs") pod "router-default-687866bb-9xvr5" (UID: "5b388e80-43fb-4367-a7c1-6c53f04caffa") : secret "router-metrics-certs-default" not found
Sep 19 11:34:11.075656 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.075343    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa857849-d4f4-4815-aa2d-8412e4a8033a-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-ftf42\" (UID: \"aa857849-d4f4-4815-aa2d-8412e4a8033a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ftf42"
Sep 19 11:34:11.075656 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.075441    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8hxv\" (UniqueName: \"kubernetes.io/projected/2066d78e-223c-4531-98aa-edc826b41c3b-kube-api-access-k8hxv\") pod \"ingress-canary-h4xjv\" (UID: \"2066d78e-223c-4531-98aa-edc826b41c3b\") " pod="openshift-ingress-canary/ingress-canary-h4xjv"
Sep 19 11:34:11.075656 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.075464    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-tmp-dir\") pod \"dns-default-t27gs\" (UID: \"7b681f4b-5ee2-4693-8e41-4be4f0a374c1\") " pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:11.075656 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.075485    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4tsq\" (UniqueName: \"kubernetes.io/projected/53ad775e-63df-4c3d-856d-f17723aed41e-kube-api-access-t4tsq\") pod \"cluster-monitoring-operator-847b8bd8d7-nll25\" (UID: \"53ad775e-63df-4c3d-856d-f17723aed41e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"
Sep 19 11:34:11.075656 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.075505    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-default-certificate\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:11.075656 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.075547    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-stats-auth\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:11.076255 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.075573    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa857849-d4f4-4815-aa2d-8412e4a8033a-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-ftf42\" (UID: \"aa857849-d4f4-4815-aa2d-8412e4a8033a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ftf42"
Sep 19 11:34:11.076255 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.075764    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:11.076255 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.075792    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cc7e845c-b6fd-4a96-8258-80b4e8f02e57-snapshots\") pod \"insights-operator-6f9c644dd9-7wgwd\" (UID: \"cc7e845c-b6fd-4a96-8258-80b4e8f02e57\") " pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:11.076255 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.076040    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60111092-bf1d-4949-b3a2-fc36e4a01d59-trusted-ca\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:11.076448 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.076267    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24aae5c7-e112-496e-ba52-a52e02e520c1-trusted-ca\") pod \"console-operator-fd4bb46c-4bxfj\" (UID: \"24aae5c7-e112-496e-ba52-a52e02e520c1\") " pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj"
Sep 19 11:34:11.076448 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.076327    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 11:34:11.076448 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.076343    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657b49657d-km2v8: secret "image-registry-tls" not found
Sep 19 11:34:11.076448 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.076402    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls podName:60111092-bf1d-4949-b3a2-fc36e4a01d59 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:11.576385051 +0000 UTC m=+35.776414832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls") pod "image-registry-657b49657d-km2v8" (UID: "60111092-bf1d-4949-b3a2-fc36e4a01d59") : secret "image-registry-tls" not found
Sep 19 11:34:11.078160 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.078117    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96c0a054-f70d-46dd-b510-9056a47a6b9b-serving-cert\") pod \"service-ca-operator-57cdc4b768-m9jqg\" (UID: \"96c0a054-f70d-46dd-b510-9056a47a6b9b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-m9jqg"
Sep 19 11:34:11.078261 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.078234    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24aae5c7-e112-496e-ba52-a52e02e520c1-serving-cert\") pod \"console-operator-fd4bb46c-4bxfj\" (UID: \"24aae5c7-e112-496e-ba52-a52e02e520c1\") " pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj"
Sep 19 11:34:11.078358 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.078335    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60111092-bf1d-4949-b3a2-fc36e4a01d59-image-registry-private-configuration\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:11.078860 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.078833    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60111092-bf1d-4949-b3a2-fc36e4a01d59-installation-pull-secrets\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:11.078980 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.078965    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa857849-d4f4-4815-aa2d-8412e4a8033a-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-ftf42\" (UID: \"aa857849-d4f4-4815-aa2d-8412e4a8033a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ftf42"
Sep 19 11:34:11.079037 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.079019    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-stats-auth\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:11.079075 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.079063    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-default-certificate\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:11.081933 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.081877    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9h65\" (UniqueName: \"kubernetes.io/projected/24aae5c7-e112-496e-ba52-a52e02e520c1-kube-api-access-f9h65\") pod \"console-operator-fd4bb46c-4bxfj\" (UID: \"24aae5c7-e112-496e-ba52-a52e02e520c1\") " pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj"
Sep 19 11:34:11.083738 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.083711    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-bound-sa-token\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:11.083928 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.083894    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2psp\" (UniqueName: \"kubernetes.io/projected/aa857849-d4f4-4815-aa2d-8412e4a8033a-kube-api-access-s2psp\") pod \"kube-storage-version-migrator-operator-55889f7b47-ftf42\" (UID: \"aa857849-d4f4-4815-aa2d-8412e4a8033a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ftf42"
Sep 19 11:34:11.084259 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.084234    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjmxg\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-kube-api-access-bjmxg\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:11.085494 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.084889    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpm6l\" (UniqueName: \"kubernetes.io/projected/40987fee-2f7a-4681-a32c-22aaa7b8a8f7-kube-api-access-hpm6l\") pod \"volume-data-source-validator-5c4db8bdf9-9zcpw\" (UID: \"40987fee-2f7a-4681-a32c-22aaa7b8a8f7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9zcpw"
Sep 19 11:34:11.085699 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.085677    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78jrp\" (UniqueName: \"kubernetes.io/projected/5b388e80-43fb-4367-a7c1-6c53f04caffa-kube-api-access-78jrp\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:11.085904 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.085884    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spblr\" (UniqueName: \"kubernetes.io/projected/96c0a054-f70d-46dd-b510-9056a47a6b9b-kube-api-access-spblr\") pod \"service-ca-operator-57cdc4b768-m9jqg\" (UID: \"96c0a054-f70d-46dd-b510-9056a47a6b9b\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-m9jqg"
Sep 19 11:34:11.086101 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.086085    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4tsq\" (UniqueName: \"kubernetes.io/projected/53ad775e-63df-4c3d-856d-f17723aed41e-kube-api-access-t4tsq\") pod \"cluster-monitoring-operator-847b8bd8d7-nll25\" (UID: \"53ad775e-63df-4c3d-856d-f17723aed41e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"
Sep 19 11:34:11.120378 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.120343    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:11.176904 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.176816    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/20620a5d-5b06-46ca-bff8-5322e24c32b2-klusterlet-config\") pod \"klusterlet-addon-workmgr-5774c959c7-6f759\" (UID: \"20620a5d-5b06-46ca-bff8-5322e24c32b2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5774c959c7-6f759"
Sep 19 11:34:11.176904 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.176857    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20620a5d-5b06-46ca-bff8-5322e24c32b2-tmp\") pod \"klusterlet-addon-workmgr-5774c959c7-6f759\" (UID: \"20620a5d-5b06-46ca-bff8-5322e24c32b2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5774c959c7-6f759"
Sep 19 11:34:11.176904 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.176885    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph8rl\" (UniqueName: \"kubernetes.io/projected/20620a5d-5b06-46ca-bff8-5322e24c32b2-kube-api-access-ph8rl\") pod \"klusterlet-addon-workmgr-5774c959c7-6f759\" (UID: \"20620a5d-5b06-46ca-bff8-5322e24c32b2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5774c959c7-6f759"
Sep 19 11:34:11.177195 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.177056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc7e845c-b6fd-4a96-8258-80b4e8f02e57-tmp\") pod \"insights-operator-6f9c644dd9-7wgwd\" (UID: \"cc7e845c-b6fd-4a96-8258-80b4e8f02e57\") " pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:11.177195 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.177084    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/83c3531f-3ddd-45b9-91cd-3ac141ebd487-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-r6svd\" (UID: \"83c3531f-3ddd-45b9-91cd-3ac141ebd487\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-r6svd"
Sep 19 11:34:11.177195 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.177102    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls\") pod \"dns-default-t27gs\" (UID: \"7b681f4b-5ee2-4693-8e41-4be4f0a374c1\") " pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:11.177195 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.177130    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc7e845c-b6fd-4a96-8258-80b4e8f02e57-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-7wgwd\" (UID: \"cc7e845c-b6fd-4a96-8258-80b4e8f02e57\") " pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:11.177389 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.177148    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-q9gbk\" (UID: \"e153e212-b7ce-4fea-89bd-970b816f0763\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk"
Sep 19 11:34:11.177658 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.177635    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc7e845c-b6fd-4a96-8258-80b4e8f02e57-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-7wgwd\" (UID: \"cc7e845c-b6fd-4a96-8258-80b4e8f02e57\") " pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:11.177784 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.177682    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c323067d-ee96-4a29-8322-83ce5f3f454f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cc5f998dc-9cvsg\" (UID: \"c323067d-ee96-4a29-8322-83ce5f3f454f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:11.177784 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.177706    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc7e845c-b6fd-4a96-8258-80b4e8f02e57-serving-cert\") pod \"insights-operator-6f9c644dd9-7wgwd\" (UID: \"cc7e845c-b6fd-4a96-8258-80b4e8f02e57\") " pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:11.177784 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.177713    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20620a5d-5b06-46ca-bff8-5322e24c32b2-tmp\") pod \"klusterlet-addon-workmgr-5774c959c7-6f759\" (UID: \"20620a5d-5b06-46ca-bff8-5322e24c32b2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5774c959c7-6f759"
Sep 19 11:34:11.177784 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.177742    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-config-volume\") pod \"dns-default-t27gs\" (UID: \"7b681f4b-5ee2-4693-8e41-4be4f0a374c1\") " pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:11.177987 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.177818    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c323067d-ee96-4a29-8322-83ce5f3f454f-ca\") pod \"cluster-proxy-proxy-agent-6cc5f998dc-9cvsg\" (UID: \"c323067d-ee96-4a29-8322-83ce5f3f454f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:11.177987 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.177882    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrslg\" (UniqueName: \"kubernetes.io/projected/1936edc6-ecd0-46ca-9b8e-08cc8b2dc383-kube-api-access-hrslg\") pod \"network-check-source-7cbc45fddc-hlvdj\" (UID: \"1936edc6-ecd0-46ca-9b8e-08cc8b2dc383\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hlvdj"
Sep 19 11:34:11.177987 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.177980    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8hxv\" (UniqueName: \"kubernetes.io/projected/2066d78e-223c-4531-98aa-edc826b41c3b-kube-api-access-k8hxv\") pod \"ingress-canary-h4xjv\" (UID: \"2066d78e-223c-4531-98aa-edc826b41c3b\") " pod="openshift-ingress-canary/ingress-canary-h4xjv"
Sep 19 11:34:11.178136 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.178012    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-tmp-dir\") pod \"dns-default-t27gs\" (UID: \"7b681f4b-5ee2-4693-8e41-4be4f0a374c1\") " pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:11.178136 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.178086    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cc7e845c-b6fd-4a96-8258-80b4e8f02e57-snapshots\") pod \"insights-operator-6f9c644dd9-7wgwd\" (UID: \"cc7e845c-b6fd-4a96-8258-80b4e8f02e57\") " pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:11.178228 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.178144    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gst2x\" (UniqueName: \"kubernetes.io/projected/c323067d-ee96-4a29-8322-83ce5f3f454f-kube-api-access-gst2x\") pod \"cluster-proxy-proxy-agent-6cc5f998dc-9cvsg\" (UID: \"c323067d-ee96-4a29-8322-83ce5f3f454f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:11.178228 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.178195    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sjwt\" (UniqueName: \"kubernetes.io/projected/36391cd8-d035-4be4-bcb3-f33a076dd18a-kube-api-access-2sjwt\") pod \"managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm\" (UID: \"36391cd8-d035-4be4-bcb3-f33a076dd18a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm"
Sep 19 11:34:11.178323 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.178242    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jdq2\" (UniqueName: \"kubernetes.io/projected/cc7e845c-b6fd-4a96-8258-80b4e8f02e57-kube-api-access-7jdq2\") pod \"insights-operator-6f9c644dd9-7wgwd\" (UID: \"cc7e845c-b6fd-4a96-8258-80b4e8f02e57\") " pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:11.178323 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.178299    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc7e845c-b6fd-4a96-8258-80b4e8f02e57-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-7wgwd\" (UID: \"cc7e845c-b6fd-4a96-8258-80b4e8f02e57\") " pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:11.178410 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.178315    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgmzc\" (UniqueName: \"kubernetes.io/projected/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-kube-api-access-qgmzc\") pod \"dns-default-t27gs\" (UID: \"7b681f4b-5ee2-4693-8e41-4be4f0a374c1\") " pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:11.178410 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.178390    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-config-volume\") pod \"dns-default-t27gs\" (UID: \"7b681f4b-5ee2-4693-8e41-4be4f0a374c1\") " pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:11.178506 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.178396    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 11:34:11.178581 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.178553    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-tmp-dir\") pod \"dns-default-t27gs\" (UID: \"7b681f4b-5ee2-4693-8e41-4be4f0a374c1\") " pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:11.178678 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.178584    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c323067d-ee96-4a29-8322-83ce5f3f454f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cc5f998dc-9cvsg\" (UID: \"c323067d-ee96-4a29-8322-83ce5f3f454f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:11.178678 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.178661    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c323067d-ee96-4a29-8322-83ce5f3f454f-hub\") pod \"cluster-proxy-proxy-agent-6cc5f998dc-9cvsg\" (UID: \"c323067d-ee96-4a29-8322-83ce5f3f454f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:11.178678 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.178665    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/83c3531f-3ddd-45b9-91cd-3ac141ebd487-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-r6svd\" (UID: \"83c3531f-3ddd-45b9-91cd-3ac141ebd487\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-r6svd"
Sep 19 11:34:11.178840 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.178694    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36391cd8-d035-4be4-bcb3-f33a076dd18a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm\" (UID: \"36391cd8-d035-4be4-bcb3-f33a076dd18a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm"
Sep 19 11:34:11.178840 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.178726    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-r6svd\" (UID: \"83c3531f-3ddd-45b9-91cd-3ac141ebd487\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-r6svd"
Sep 19 11:34:11.179254 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.178695    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc7e845c-b6fd-4a96-8258-80b4e8f02e57-tmp\") pod \"insights-operator-6f9c644dd9-7wgwd\" (UID: \"cc7e845c-b6fd-4a96-8258-80b4e8f02e57\") " pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:11.181204 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.179525    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls podName:7b681f4b-5ee2-4693-8e41-4be4f0a374c1 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:11.67950384 +0000 UTC m=+35.879533633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls") pod "dns-default-t27gs" (UID: "7b681f4b-5ee2-4693-8e41-4be4f0a374c1") : secret "dns-default-metrics-tls" not found
Sep 19 11:34:11.181204 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.179949    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c323067d-ee96-4a29-8322-83ce5f3f454f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cc5f998dc-9cvsg\" (UID: \"c323067d-ee96-4a29-8322-83ce5f3f454f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:11.181204 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.180024    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 11:34:11.181204 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.180182    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert podName:83c3531f-3ddd-45b9-91cd-3ac141ebd487 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:11.680099864 +0000 UTC m=+35.880129651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-r6svd" (UID: "83c3531f-3ddd-45b9-91cd-3ac141ebd487") : secret "networking-console-plugin-cert" not found
Sep 19 11:34:11.181204 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.180292    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cc7e845c-b6fd-4a96-8258-80b4e8f02e57-snapshots\") pod \"insights-operator-6f9c644dd9-7wgwd\" (UID: \"cc7e845c-b6fd-4a96-8258-80b4e8f02e57\") " pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:11.181204 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.180476    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2066d78e-223c-4531-98aa-edc826b41c3b-cert\") pod \"ingress-canary-h4xjv\" (UID: \"2066d78e-223c-4531-98aa-edc826b41c3b\") " pod="openshift-ingress-canary/ingress-canary-h4xjv"
Sep 19 11:34:11.181204 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.180518    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4vtk\" (UniqueName: \"kubernetes.io/projected/e153e212-b7ce-4fea-89bd-970b816f0763-kube-api-access-v4vtk\") pod \"cluster-samples-operator-685d7d68b5-q9gbk\" (UID: \"e153e212-b7ce-4fea-89bd-970b816f0763\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk"
Sep 19 11:34:11.181204 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.180764    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 11:34:11.181204 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.181139    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/20620a5d-5b06-46ca-bff8-5322e24c32b2-klusterlet-config\") pod \"klusterlet-addon-workmgr-5774c959c7-6f759\" (UID: \"20620a5d-5b06-46ca-bff8-5322e24c32b2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5774c959c7-6f759"
Sep 19 11:34:11.181707 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.181546    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls podName:e153e212-b7ce-4fea-89bd-970b816f0763 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:11.680796677 +0000 UTC m=+35.880826459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-q9gbk" (UID: "e153e212-b7ce-4fea-89bd-970b816f0763") : secret "samples-operator-tls" not found
Sep 19 11:34:11.184469 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.182553    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c323067d-ee96-4a29-8322-83ce5f3f454f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cc5f998dc-9cvsg\" (UID: \"c323067d-ee96-4a29-8322-83ce5f3f454f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:11.184469 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.182764    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc7e845c-b6fd-4a96-8258-80b4e8f02e57-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-7wgwd\" (UID: \"cc7e845c-b6fd-4a96-8258-80b4e8f02e57\") " pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:11.185209 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.185179    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc7e845c-b6fd-4a96-8258-80b4e8f02e57-serving-cert\") pod \"insights-operator-6f9c644dd9-7wgwd\" (UID: \"cc7e845c-b6fd-4a96-8258-80b4e8f02e57\") " pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:11.187387 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.187366    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c323067d-ee96-4a29-8322-83ce5f3f454f-hub\") pod \"cluster-proxy-proxy-agent-6cc5f998dc-9cvsg\" (UID: \"c323067d-ee96-4a29-8322-83ce5f3f454f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:11.189558 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.189307    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:11.189558 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.189513    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36391cd8-d035-4be4-bcb3-f33a076dd18a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm\" (UID: \"36391cd8-d035-4be4-bcb3-f33a076dd18a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm"
Sep 19 11:34:11.190259 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.190220    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c323067d-ee96-4a29-8322-83ce5f3f454f-ca\") pod \"cluster-proxy-proxy-agent-6cc5f998dc-9cvsg\" (UID: \"c323067d-ee96-4a29-8322-83ce5f3f454f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:11.190531 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.190370    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2066d78e-223c-4531-98aa-edc826b41c3b-cert\") pod \"ingress-canary-h4xjv\" (UID: \"2066d78e-223c-4531-98aa-edc826b41c3b\") " pod="openshift-ingress-canary/ingress-canary-h4xjv"
Sep 19 11:34:11.191491 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.191427    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c323067d-ee96-4a29-8322-83ce5f3f454f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cc5f998dc-9cvsg\" (UID: \"c323067d-ee96-4a29-8322-83ce5f3f454f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:11.191491 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.191458    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jdq2\" (UniqueName: \"kubernetes.io/projected/cc7e845c-b6fd-4a96-8258-80b4e8f02e57-kube-api-access-7jdq2\") pod \"insights-operator-6f9c644dd9-7wgwd\" (UID: \"cc7e845c-b6fd-4a96-8258-80b4e8f02e57\") " pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:11.191640 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.191589    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c323067d-ee96-4a29-8322-83ce5f3f454f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cc5f998dc-9cvsg\" (UID: \"c323067d-ee96-4a29-8322-83ce5f3f454f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:11.192451 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.192430    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph8rl\" (UniqueName: \"kubernetes.io/projected/20620a5d-5b06-46ca-bff8-5322e24c32b2-kube-api-access-ph8rl\") pod \"klusterlet-addon-workmgr-5774c959c7-6f759\" (UID: \"20620a5d-5b06-46ca-bff8-5322e24c32b2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5774c959c7-6f759"
Sep 19 11:34:11.209427 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.209392    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gst2x\" (UniqueName: \"kubernetes.io/projected/c323067d-ee96-4a29-8322-83ce5f3f454f-kube-api-access-gst2x\") pod \"cluster-proxy-proxy-agent-6cc5f998dc-9cvsg\" (UID: \"c323067d-ee96-4a29-8322-83ce5f3f454f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:11.218436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.218405    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-m9jqg"
Sep 19 11:34:11.228433 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.228395    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sjwt\" (UniqueName: \"kubernetes.io/projected/36391cd8-d035-4be4-bcb3-f33a076dd18a-kube-api-access-2sjwt\") pod \"managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm\" (UID: \"36391cd8-d035-4be4-bcb3-f33a076dd18a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm"
Sep 19 11:34:11.244459 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.244414    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj"
Sep 19 11:34:11.247964 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.247938    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgmzc\" (UniqueName: \"kubernetes.io/projected/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-kube-api-access-qgmzc\") pod \"dns-default-t27gs\" (UID: \"7b681f4b-5ee2-4693-8e41-4be4f0a374c1\") " pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:11.268634 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.268603    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4vtk\" (UniqueName: \"kubernetes.io/projected/e153e212-b7ce-4fea-89bd-970b816f0763-kube-api-access-v4vtk\") pod \"cluster-samples-operator-685d7d68b5-q9gbk\" (UID: \"e153e212-b7ce-4fea-89bd-970b816f0763\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk"
Sep 19 11:34:11.280439 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.280408    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9zcpw"
Sep 19 11:34:11.287445 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.287272    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ftf42"
Sep 19 11:34:11.287712 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.287687    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8hxv\" (UniqueName: \"kubernetes.io/projected/2066d78e-223c-4531-98aa-edc826b41c3b-kube-api-access-k8hxv\") pod \"ingress-canary-h4xjv\" (UID: \"2066d78e-223c-4531-98aa-edc826b41c3b\") " pod="openshift-ingress-canary/ingress-canary-h4xjv"
Sep 19 11:34:11.306995 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.306974    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrslg\" (UniqueName: \"kubernetes.io/projected/1936edc6-ecd0-46ca-9b8e-08cc8b2dc383-kube-api-access-hrslg\") pod \"network-check-source-7cbc45fddc-hlvdj\" (UID: \"1936edc6-ecd0-46ca-9b8e-08cc8b2dc383\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hlvdj"
Sep 19 11:34:11.308773 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.308747    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd"
Sep 19 11:34:11.316215 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.316198    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h4xjv"
Sep 19 11:34:11.324376 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.324346    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hlvdj"
Sep 19 11:34:11.331653 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.331608    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5774c959c7-6f759"
Sep 19 11:34:11.361139 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.361053    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:34:11.367132 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.367095    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm"
Sep 19 11:34:11.526847 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.526772    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:11.585063 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.584495    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:11.585063 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.584551    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nll25\" (UID: \"53ad775e-63df-4c3d-856d-f17723aed41e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"
Sep 19 11:34:11.585063 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.584594    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs\") pod \"network-metrics-daemon-pgph6\" (UID: \"83707003-2551-42b5-8943-8243dd6daa80\") " pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:34:11.585063 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.584689    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:11.585063 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.584783    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:11.585063 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.584936    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 11:34:11.585063 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.584995    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 11:34:11.585063 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.585004    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs podName:5b388e80-43fb-4367-a7c1-6c53f04caffa nodeName:}" failed. No retries permitted until 2026-09-19 11:34:12.584980895 +0000 UTC m=+36.785010690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs") pod "router-default-687866bb-9xvr5" (UID: "5b388e80-43fb-4367-a7c1-6c53f04caffa") : secret "router-metrics-certs-default" not found
Sep 19 11:34:11.585063 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.585045    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls podName:53ad775e-63df-4c3d-856d-f17723aed41e nodeName:}" failed. No retries permitted until 2026-09-19 11:34:12.585034321 +0000 UTC m=+36.785064101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-nll25" (UID: "53ad775e-63df-4c3d-856d-f17723aed41e") : secret "cluster-monitoring-operator-tls" not found
Sep 19 11:34:11.585462 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.585077    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 11:34:11.585462 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.585096    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657b49657d-km2v8: secret "image-registry-tls" not found
Sep 19 11:34:11.585462 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.585108    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle podName:5b388e80-43fb-4367-a7c1-6c53f04caffa nodeName:}" failed. No retries permitted until 2026-09-19 11:34:12.585098681 +0000 UTC m=+36.785128465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle") pod "router-default-687866bb-9xvr5" (UID: "5b388e80-43fb-4367-a7c1-6c53f04caffa") : configmap references non-existent config key: service-ca.crt
Sep 19 11:34:11.585462 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.585151    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls podName:60111092-bf1d-4949-b3a2-fc36e4a01d59 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:12.585134079 +0000 UTC m=+36.785163875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls") pod "image-registry-657b49657d-km2v8" (UID: "60111092-bf1d-4949-b3a2-fc36e4a01d59") : secret "image-registry-tls" not found
Sep 19 11:34:11.585462 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.585160    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 11:34:11.585462 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.585187    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs podName:83707003-2551-42b5-8943-8243dd6daa80 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:43.585177909 +0000 UTC m=+67.785207689 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs") pod "network-metrics-daemon-pgph6" (UID: "83707003-2551-42b5-8943-8243dd6daa80") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 11:34:11.620326 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.620283    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:11.647624 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.647509    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-4bxfj"]
Sep 19 11:34:11.649223 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.649113    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-m9jqg"]
Sep 19 11:34:11.650381 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.650349    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ftf42"]
Sep 19 11:34:11.650969 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:34:11.650939    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24aae5c7_e112_496e_ba52_a52e02e520c1.slice/crio-858a3079a500f77e32aeb832c8e34650c76a5d2633e30d18413ea87dff13cd25 WatchSource:0}: Error finding container 858a3079a500f77e32aeb832c8e34650c76a5d2633e30d18413ea87dff13cd25: Status 404 returned error can't find the container with id 858a3079a500f77e32aeb832c8e34650c76a5d2633e30d18413ea87dff13cd25
Sep 19 11:34:11.651326 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:34:11.651306    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c0a054_f70d_46dd_b510_9056a47a6b9b.slice/crio-491eed5bda477eb1a9b5eafcabbc80abc119f5331d448d384fd1d5420b9b01d8 WatchSource:0}: Error finding container 491eed5bda477eb1a9b5eafcabbc80abc119f5331d448d384fd1d5420b9b01d8: Status 404 returned error can't find the container with id 491eed5bda477eb1a9b5eafcabbc80abc119f5331d448d384fd1d5420b9b01d8
Sep 19 11:34:11.655733 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.655710    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9zcpw"]
Sep 19 11:34:11.660714 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:34:11.660682    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40987fee_2f7a_4681_a32c_22aaa7b8a8f7.slice/crio-625acb10676418728d5565a7dc390c107b6de941e2a5df8ba4c4b495f7d127a6 WatchSource:0}: Error finding container 625acb10676418728d5565a7dc390c107b6de941e2a5df8ba4c4b495f7d127a6: Status 404 returned error can't find the container with id 625acb10676418728d5565a7dc390c107b6de941e2a5df8ba4c4b495f7d127a6
Sep 19 11:34:11.677192 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.677155    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5774c959c7-6f759"]
Sep 19 11:34:11.680334 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:34:11.680304    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20620a5d_5b06_46ca_bff8_5322e24c32b2.slice/crio-0c707066f1a2705d26dac2f51a4e211f3b0f867464931dc95956107921063f2e WatchSource:0}: Error finding container 0c707066f1a2705d26dac2f51a4e211f3b0f867464931dc95956107921063f2e: Status 404 returned error can't find the container with id 0c707066f1a2705d26dac2f51a4e211f3b0f867464931dc95956107921063f2e
Sep 19 11:34:11.685247 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.685222    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-r6svd\" (UID: \"83c3531f-3ddd-45b9-91cd-3ac141ebd487\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-r6svd"
Sep 19 11:34:11.685418 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.685278    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7wwl\" (UniqueName: \"kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl\") pod \"network-check-target-m2dhj\" (UID: \"d0c272c9-c396-486e-b63b-6726cf1321d8\") " pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:34:11.685418 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.685326    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls\") pod \"dns-default-t27gs\" (UID: \"7b681f4b-5ee2-4693-8e41-4be4f0a374c1\") " pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:11.685418 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.685346    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-q9gbk\" (UID: \"e153e212-b7ce-4fea-89bd-970b816f0763\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk"
Sep 19 11:34:11.685418 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.685367    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret\") pod \"global-pull-secret-syncer-dpgkd\" (UID: \"f96896a2-5494-4bc6-81c9-154860ee5f6b\") " pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:34:11.685418 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.685381    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 11:34:11.685656 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.685447    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert podName:83c3531f-3ddd-45b9-91cd-3ac141ebd487 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:12.685427281 +0000 UTC m=+36.885457074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-r6svd" (UID: "83c3531f-3ddd-45b9-91cd-3ac141ebd487") : secret "networking-console-plugin-cert" not found
Sep 19 11:34:11.685656 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.685522    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 11:34:11.685656 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.685554    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls podName:7b681f4b-5ee2-4693-8e41-4be4f0a374c1 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:12.685543238 +0000 UTC m=+36.885573032 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls") pod "dns-default-t27gs" (UID: "7b681f4b-5ee2-4693-8e41-4be4f0a374c1") : secret "dns-default-metrics-tls" not found
Sep 19 11:34:11.685975 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.685955    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 11:34:11.686061 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:11.686016    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls podName:e153e212-b7ce-4fea-89bd-970b816f0763 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:12.686000698 +0000 UTC m=+36.886030478 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-q9gbk" (UID: "e153e212-b7ce-4fea-89bd-970b816f0763") : secret "samples-operator-tls" not found
Sep 19 11:34:11.688299 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.688262    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f96896a2-5494-4bc6-81c9-154860ee5f6b-original-pull-secret\") pod \"global-pull-secret-syncer-dpgkd\" (UID: \"f96896a2-5494-4bc6-81c9-154860ee5f6b\") " pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:34:11.688595 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.688570    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7wwl\" (UniqueName: \"kubernetes.io/projected/d0c272c9-c396-486e-b63b-6726cf1321d8-kube-api-access-w7wwl\") pod \"network-check-target-m2dhj\" (UID: \"d0c272c9-c396-486e-b63b-6726cf1321d8\") " pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:34:11.730698 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.730580    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-7wgwd"]
Sep 19 11:34:11.733113 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:34:11.733089    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7e845c_b6fd_4a96_8258_80b4e8f02e57.slice/crio-b20c5eb9dc888153a38b7215c407469571da9f193b15cf10227762c405533e7f WatchSource:0}: Error finding container b20c5eb9dc888153a38b7215c407469571da9f193b15cf10227762c405533e7f: Status 404 returned error can't find the container with id b20c5eb9dc888153a38b7215c407469571da9f193b15cf10227762c405533e7f
Sep 19 11:34:11.760601 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.760560    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h4xjv"]
Sep 19 11:34:11.761731 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.761707    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-hlvdj"]
Sep 19 11:34:11.763297 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:34:11.763263    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2066d78e_223c_4531_98aa_edc826b41c3b.slice/crio-8046ebe5beab4d5397efd4c9de05778b6a9b13dd2c339c1169299fc822201568 WatchSource:0}: Error finding container 8046ebe5beab4d5397efd4c9de05778b6a9b13dd2c339c1169299fc822201568: Status 404 returned error can't find the container with id 8046ebe5beab4d5397efd4c9de05778b6a9b13dd2c339c1169299fc822201568
Sep 19 11:34:11.764103 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:34:11.764065    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1936edc6_ecd0_46ca_9b8e_08cc8b2dc383.slice/crio-a75701adfa7813d7a0b04c91dc15fe7c00644c2a6126727980c72a34f2c83d8e WatchSource:0}: Error finding container a75701adfa7813d7a0b04c91dc15fe7c00644c2a6126727980c72a34f2c83d8e: Status 404 returned error can't find the container with id a75701adfa7813d7a0b04c91dc15fe7c00644c2a6126727980c72a34f2c83d8e
Sep 19 11:34:11.789155 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.789124    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"]
Sep 19 11:34:11.792532 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:34:11.792497    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc323067d_ee96_4a29_8322_83ce5f3f454f.slice/crio-d2609cb2b8084d07f78285cf14395a86444caef10399f27c6ce05e2bab88c2a6 WatchSource:0}: Error finding container d2609cb2b8084d07f78285cf14395a86444caef10399f27c6ce05e2bab88c2a6: Status 404 returned error can't find the container with id d2609cb2b8084d07f78285cf14395a86444caef10399f27c6ce05e2bab88c2a6
Sep 19 11:34:11.798762 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.798732    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm"]
Sep 19 11:34:11.803829 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:34:11.803797    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36391cd8_d035_4be4_bcb3_f33a076dd18a.slice/crio-70da2b03fd3cf260de221843f3aa5ad34f3ed4a91a7ed8f4410fb9a41129cef6 WatchSource:0}: Error finding container 70da2b03fd3cf260de221843f3aa5ad34f3ed4a91a7ed8f4410fb9a41129cef6: Status 404 returned error can't find the container with id 70da2b03fd3cf260de221843f3aa5ad34f3ed4a91a7ed8f4410fb9a41129cef6
Sep 19 11:34:11.812262 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.812236    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dpgkd"
Sep 19 11:34:11.833204 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.833172    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:34:11.897097 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.897068    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:34:11.901617 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.900819    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6c6f7\""
Sep 19 11:34:11.901617 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:11.901108    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 11:34:12.101708 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.101540    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hlvdj" event={"ID":"1936edc6-ecd0-46ca-9b8e-08cc8b2dc383","Type":"ContainerStarted","Data":"a75701adfa7813d7a0b04c91dc15fe7c00644c2a6126727980c72a34f2c83d8e"}
Sep 19 11:34:12.102830 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.102800    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5774c959c7-6f759" event={"ID":"20620a5d-5b06-46ca-bff8-5322e24c32b2","Type":"ContainerStarted","Data":"0c707066f1a2705d26dac2f51a4e211f3b0f867464931dc95956107921063f2e"}
Sep 19 11:34:12.104073 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.104048    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h4xjv" event={"ID":"2066d78e-223c-4531-98aa-edc826b41c3b","Type":"ContainerStarted","Data":"8046ebe5beab4d5397efd4c9de05778b6a9b13dd2c339c1169299fc822201568"}
Sep 19 11:34:12.105240 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.105217    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9zcpw" event={"ID":"40987fee-2f7a-4681-a32c-22aaa7b8a8f7","Type":"ContainerStarted","Data":"625acb10676418728d5565a7dc390c107b6de941e2a5df8ba4c4b495f7d127a6"}
Sep 19 11:34:12.106290 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.106264    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg" event={"ID":"c323067d-ee96-4a29-8322-83ce5f3f454f","Type":"ContainerStarted","Data":"d2609cb2b8084d07f78285cf14395a86444caef10399f27c6ce05e2bab88c2a6"}
Sep 19 11:34:12.107788 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.107764    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ftf42" event={"ID":"aa857849-d4f4-4815-aa2d-8412e4a8033a","Type":"ContainerStarted","Data":"4853b0ab802349864c9b61a5434d400ac75f7797fb67da13112cc04709f7e333"}
Sep 19 11:34:12.108512 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.108492    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dpgkd"]
Sep 19 11:34:12.109094 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.109059    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj" event={"ID":"24aae5c7-e112-496e-ba52-a52e02e520c1","Type":"ContainerStarted","Data":"858a3079a500f77e32aeb832c8e34650c76a5d2633e30d18413ea87dff13cd25"}
Sep 19 11:34:12.110421 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.110397    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm" event={"ID":"36391cd8-d035-4be4-bcb3-f33a076dd18a","Type":"ContainerStarted","Data":"70da2b03fd3cf260de221843f3aa5ad34f3ed4a91a7ed8f4410fb9a41129cef6"}
Sep 19 11:34:12.111878 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.111855    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd" event={"ID":"cc7e845c-b6fd-4a96-8258-80b4e8f02e57","Type":"ContainerStarted","Data":"b20c5eb9dc888153a38b7215c407469571da9f193b15cf10227762c405533e7f"}
Sep 19 11:34:12.112251 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:34:12.112221    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96896a2_5494_4bc6_81c9_154860ee5f6b.slice/crio-9f0fdca4c2e59e3a9751d5a90c0f16923c194554b7c2bf47c6e16de59974f34a WatchSource:0}: Error finding container 9f0fdca4c2e59e3a9751d5a90c0f16923c194554b7c2bf47c6e16de59974f34a: Status 404 returned error can't find the container with id 9f0fdca4c2e59e3a9751d5a90c0f16923c194554b7c2bf47c6e16de59974f34a
Sep 19 11:34:12.113134 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.113114    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-m9jqg" event={"ID":"96c0a054-f70d-46dd-b510-9056a47a6b9b","Type":"ContainerStarted","Data":"491eed5bda477eb1a9b5eafcabbc80abc119f5331d448d384fd1d5420b9b01d8"}
Sep 19 11:34:12.130453 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.130417    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m2dhj"]
Sep 19 11:34:12.134304 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:34:12.134272    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c272c9_c396_486e_b63b_6726cf1321d8.slice/crio-02db2059a28618433e32d2983314c6b75da0f2a5685c4c6aa27ee78d392b2024 WatchSource:0}: Error finding container 02db2059a28618433e32d2983314c6b75da0f2a5685c4c6aa27ee78d392b2024: Status 404 returned error can't find the container with id 02db2059a28618433e32d2983314c6b75da0f2a5685c4c6aa27ee78d392b2024
Sep 19 11:34:12.325469 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.325420    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:12.595623 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.595580    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:12.595824 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.595697    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:12.595824 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.595760    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:12.595824 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.595793    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nll25\" (UID: \"53ad775e-63df-4c3d-856d-f17723aed41e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"
Sep 19 11:34:12.596030 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:12.596018    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 11:34:12.596099 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:12.596085    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls podName:53ad775e-63df-4c3d-856d-f17723aed41e nodeName:}" failed. No retries permitted until 2026-09-19 11:34:14.596066565 +0000 UTC m=+38.796096351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-nll25" (UID: "53ad775e-63df-4c3d-856d-f17723aed41e") : secret "cluster-monitoring-operator-tls" not found
Sep 19 11:34:12.596578 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:12.596561    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle podName:5b388e80-43fb-4367-a7c1-6c53f04caffa nodeName:}" failed. No retries permitted until 2026-09-19 11:34:14.596534021 +0000 UTC m=+38.796563801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle") pod "router-default-687866bb-9xvr5" (UID: "5b388e80-43fb-4367-a7c1-6c53f04caffa") : configmap references non-existent config key: service-ca.crt
Sep 19 11:34:12.596662 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:12.596647    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 11:34:12.596709 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:12.596686    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs podName:5b388e80-43fb-4367-a7c1-6c53f04caffa nodeName:}" failed. No retries permitted until 2026-09-19 11:34:14.596673486 +0000 UTC m=+38.796703266 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs") pod "router-default-687866bb-9xvr5" (UID: "5b388e80-43fb-4367-a7c1-6c53f04caffa") : secret "router-metrics-certs-default" not found
Sep 19 11:34:12.596762 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:12.596753    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 11:34:12.596806 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:12.596763    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657b49657d-km2v8: secret "image-registry-tls" not found
Sep 19 11:34:12.596806 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:12.596794    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls podName:60111092-bf1d-4949-b3a2-fc36e4a01d59 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:14.596782771 +0000 UTC m=+38.796812562 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls") pod "image-registry-657b49657d-km2v8" (UID: "60111092-bf1d-4949-b3a2-fc36e4a01d59") : secret "image-registry-tls" not found
Sep 19 11:34:12.696612 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.696568    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-q9gbk\" (UID: \"e153e212-b7ce-4fea-89bd-970b816f0763\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk"
Sep 19 11:34:12.696809 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:12.696729    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 11:34:12.696809 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.696766    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-r6svd\" (UID: \"83c3531f-3ddd-45b9-91cd-3ac141ebd487\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-r6svd"
Sep 19 11:34:12.696809 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:12.696800    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls podName:e153e212-b7ce-4fea-89bd-970b816f0763 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:14.696781255 +0000 UTC m=+38.896811035 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-q9gbk" (UID: "e153e212-b7ce-4fea-89bd-970b816f0763") : secret "samples-operator-tls" not found
Sep 19 11:34:12.697009 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:12.696830    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 11:34:12.697009 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:12.696866    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert podName:83c3531f-3ddd-45b9-91cd-3ac141ebd487 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:14.696854044 +0000 UTC m=+38.896883838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-r6svd" (UID: "83c3531f-3ddd-45b9-91cd-3ac141ebd487") : secret "networking-console-plugin-cert" not found
Sep 19 11:34:12.697009 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.696893    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls\") pod \"dns-default-t27gs\" (UID: \"7b681f4b-5ee2-4693-8e41-4be4f0a374c1\") " pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:12.697009 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:12.696998    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 11:34:12.697200 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:12.697030    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls podName:7b681f4b-5ee2-4693-8e41-4be4f0a374c1 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:14.697019875 +0000 UTC m=+38.897049659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls") pod "dns-default-t27gs" (UID: "7b681f4b-5ee2-4693-8e41-4be4f0a374c1") : secret "dns-default-metrics-tls" not found
Sep 19 11:34:12.734721 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.734632    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:12.791898 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:12.791766    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:13.139383 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:13.137511    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dpgkd" event={"ID":"f96896a2-5494-4bc6-81c9-154860ee5f6b","Type":"ContainerStarted","Data":"9f0fdca4c2e59e3a9751d5a90c0f16923c194554b7c2bf47c6e16de59974f34a"}
Sep 19 11:34:13.159874 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:13.159823    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m2dhj" event={"ID":"d0c272c9-c396-486e-b63b-6726cf1321d8","Type":"ContainerStarted","Data":"02db2059a28618433e32d2983314c6b75da0f2a5685c4c6aa27ee78d392b2024"}
Sep 19 11:34:13.461745 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:13.461050    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:13.528383 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:13.528342    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:13.896141 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:13.895777    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:14.620196 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:14.620137    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:14.620801 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:14.620248    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:14.620801 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:14.620290    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:14.620801 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:14.620321    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nll25\" (UID: \"53ad775e-63df-4c3d-856d-f17723aed41e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"
Sep 19 11:34:14.620801 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:14.620518    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 11:34:14.620801 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:14.620584    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls podName:53ad775e-63df-4c3d-856d-f17723aed41e nodeName:}" failed. No retries permitted until 2026-09-19 11:34:18.620563921 +0000 UTC m=+42.820593706 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-nll25" (UID: "53ad775e-63df-4c3d-856d-f17723aed41e") : secret "cluster-monitoring-operator-tls" not found
Sep 19 11:34:14.621393 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:14.621050    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle podName:5b388e80-43fb-4367-a7c1-6c53f04caffa nodeName:}" failed. No retries permitted until 2026-09-19 11:34:18.621032691 +0000 UTC m=+42.821062487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle") pod "router-default-687866bb-9xvr5" (UID: "5b388e80-43fb-4367-a7c1-6c53f04caffa") : configmap references non-existent config key: service-ca.crt
Sep 19 11:34:14.621393 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:14.621128    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 11:34:14.621393 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:14.621166    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs podName:5b388e80-43fb-4367-a7c1-6c53f04caffa nodeName:}" failed. No retries permitted until 2026-09-19 11:34:18.621153884 +0000 UTC m=+42.821183675 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs") pod "router-default-687866bb-9xvr5" (UID: "5b388e80-43fb-4367-a7c1-6c53f04caffa") : secret "router-metrics-certs-default" not found
Sep 19 11:34:14.621393 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:14.621222    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 11:34:14.621393 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:14.621232    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657b49657d-km2v8: secret "image-registry-tls" not found
Sep 19 11:34:14.621393 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:14.621262    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls podName:60111092-bf1d-4949-b3a2-fc36e4a01d59 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:18.621252342 +0000 UTC m=+42.821282128 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls") pod "image-registry-657b49657d-km2v8" (UID: "60111092-bf1d-4949-b3a2-fc36e4a01d59") : secret "image-registry-tls" not found
Sep 19 11:34:14.721420 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:14.721372    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-r6svd\" (UID: \"83c3531f-3ddd-45b9-91cd-3ac141ebd487\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-r6svd"
Sep 19 11:34:14.721604 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:14.721482    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls\") pod \"dns-default-t27gs\" (UID: \"7b681f4b-5ee2-4693-8e41-4be4f0a374c1\") " pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:14.721604 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:14.721518    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-q9gbk\" (UID: \"e153e212-b7ce-4fea-89bd-970b816f0763\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk"
Sep 19 11:34:14.721737 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:14.721679    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 11:34:14.721783 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:14.721745    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls podName:e153e212-b7ce-4fea-89bd-970b816f0763 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:18.721727191 +0000 UTC m=+42.921756977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-q9gbk" (UID: "e153e212-b7ce-4fea-89bd-970b816f0763") : secret "samples-operator-tls" not found
Sep 19 11:34:14.722184 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:14.722162    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 11:34:14.722283 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:14.722224    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert podName:83c3531f-3ddd-45b9-91cd-3ac141ebd487 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:18.722206527 +0000 UTC m=+42.922236308 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-r6svd" (UID: "83c3531f-3ddd-45b9-91cd-3ac141ebd487") : secret "networking-console-plugin-cert" not found
Sep 19 11:34:14.722283 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:14.722281    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 11:34:14.722401 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:14.722312    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls podName:7b681f4b-5ee2-4693-8e41-4be4f0a374c1 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:18.722301543 +0000 UTC m=+42.922331327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls") pod "dns-default-t27gs" (UID: "7b681f4b-5ee2-4693-8e41-4be4f0a374c1") : secret "dns-default-metrics-tls" not found
Sep 19 11:34:15.797650 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:15.797559    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:15.853767 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:15.853664    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:18.668809 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:18.668760    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:18.669256 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:18.668870    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:18.669256 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:18.668935    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:18.669256 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:18.668966    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nll25\" (UID: \"53ad775e-63df-4c3d-856d-f17723aed41e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"
Sep 19 11:34:18.669256 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:18.668996    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle podName:5b388e80-43fb-4367-a7c1-6c53f04caffa nodeName:}" failed. No retries permitted until 2026-09-19 11:34:26.668969933 +0000 UTC m=+50.868999736 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle") pod "router-default-687866bb-9xvr5" (UID: "5b388e80-43fb-4367-a7c1-6c53f04caffa") : configmap references non-existent config key: service-ca.crt
Sep 19 11:34:18.669256 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:18.669071    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 11:34:18.669256 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:18.669077    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 11:34:18.669256 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:18.669123    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls podName:53ad775e-63df-4c3d-856d-f17723aed41e nodeName:}" failed. No retries permitted until 2026-09-19 11:34:26.669106765 +0000 UTC m=+50.869136548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-nll25" (UID: "53ad775e-63df-4c3d-856d-f17723aed41e") : secret "cluster-monitoring-operator-tls" not found
Sep 19 11:34:18.669256 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:18.669151    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs podName:5b388e80-43fb-4367-a7c1-6c53f04caffa nodeName:}" failed. No retries permitted until 2026-09-19 11:34:26.669133311 +0000 UTC m=+50.869163104 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs") pod "router-default-687866bb-9xvr5" (UID: "5b388e80-43fb-4367-a7c1-6c53f04caffa") : secret "router-metrics-certs-default" not found
Sep 19 11:34:18.669256 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:18.669191    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 11:34:18.669256 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:18.669202    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657b49657d-km2v8: secret "image-registry-tls" not found
Sep 19 11:34:18.669256 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:18.669241    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls podName:60111092-bf1d-4949-b3a2-fc36e4a01d59 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:26.669229766 +0000 UTC m=+50.869259552 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls") pod "image-registry-657b49657d-km2v8" (UID: "60111092-bf1d-4949-b3a2-fc36e4a01d59") : secret "image-registry-tls" not found
Sep 19 11:34:18.770519 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:18.770470    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-r6svd\" (UID: \"83c3531f-3ddd-45b9-91cd-3ac141ebd487\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-r6svd"
Sep 19 11:34:18.770734 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:18.770584    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls\") pod \"dns-default-t27gs\" (UID: \"7b681f4b-5ee2-4693-8e41-4be4f0a374c1\") " pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:18.770734 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:18.770621    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-q9gbk\" (UID: \"e153e212-b7ce-4fea-89bd-970b816f0763\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk"
Sep 19 11:34:18.770734 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:18.770637    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 11:34:18.770734 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:18.770709    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert podName:83c3531f-3ddd-45b9-91cd-3ac141ebd487 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:26.770692436 +0000 UTC m=+50.970722220 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-r6svd" (UID: "83c3531f-3ddd-45b9-91cd-3ac141ebd487") : secret "networking-console-plugin-cert" not found
Sep 19 11:34:18.770734 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:18.770731    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 11:34:18.770961 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:18.770740    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 11:34:18.770961 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:18.770771    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls podName:e153e212-b7ce-4fea-89bd-970b816f0763 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:26.770760851 +0000 UTC m=+50.970790631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-q9gbk" (UID: "e153e212-b7ce-4fea-89bd-970b816f0763") : secret "samples-operator-tls" not found
Sep 19 11:34:18.770961 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:18.770830    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls podName:7b681f4b-5ee2-4693-8e41-4be4f0a374c1 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:26.770816681 +0000 UTC m=+50.970846465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls") pod "dns-default-t27gs" (UID: "7b681f4b-5ee2-4693-8e41-4be4f0a374c1") : secret "dns-default-metrics-tls" not found
Sep 19 11:34:22.401061 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:22.401029    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:22.457513 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:22.457475    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:26.018416 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:26.018342    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:26.117930 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:26.117878    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:26.746260 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:26.746214    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:26.746441 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:26.746285    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:26.746441 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:26.746317    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:26.746441 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:26.746335    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nll25\" (UID: \"53ad775e-63df-4c3d-856d-f17723aed41e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"
Sep 19 11:34:26.746441 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:26.746386    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 11:34:26.746441 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:26.746406    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 11:34:26.746441 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:26.746403    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle podName:5b388e80-43fb-4367-a7c1-6c53f04caffa nodeName:}" failed. No retries permitted until 2026-09-19 11:34:42.746386212 +0000 UTC m=+66.946416009 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle") pod "router-default-687866bb-9xvr5" (UID: "5b388e80-43fb-4367-a7c1-6c53f04caffa") : configmap references non-existent config key: service-ca.crt
Sep 19 11:34:26.746657 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:26.746451    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 11:34:26.746657 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:26.746460    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls podName:53ad775e-63df-4c3d-856d-f17723aed41e nodeName:}" failed. No retries permitted until 2026-09-19 11:34:42.746449541 +0000 UTC m=+66.946479344 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-nll25" (UID: "53ad775e-63df-4c3d-856d-f17723aed41e") : secret "cluster-monitoring-operator-tls" not found
Sep 19 11:34:26.746657 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:26.746467    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-657b49657d-km2v8: secret "image-registry-tls" not found
Sep 19 11:34:26.746657 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:26.746481    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs podName:5b388e80-43fb-4367-a7c1-6c53f04caffa nodeName:}" failed. No retries permitted until 2026-09-19 11:34:42.746473095 +0000 UTC m=+66.946502875 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs") pod "router-default-687866bb-9xvr5" (UID: "5b388e80-43fb-4367-a7c1-6c53f04caffa") : secret "router-metrics-certs-default" not found
Sep 19 11:34:26.746657 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:26.746519    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls podName:60111092-bf1d-4949-b3a2-fc36e4a01d59 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:42.746507395 +0000 UTC m=+66.946537176 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls") pod "image-registry-657b49657d-km2v8" (UID: "60111092-bf1d-4949-b3a2-fc36e4a01d59") : secret "image-registry-tls" not found
Sep 19 11:34:26.847532 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:26.847499    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls\") pod \"dns-default-t27gs\" (UID: \"7b681f4b-5ee2-4693-8e41-4be4f0a374c1\") " pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:26.847616 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:26.847565    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-q9gbk\" (UID: \"e153e212-b7ce-4fea-89bd-970b816f0763\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk"
Sep 19 11:34:26.847658 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:26.847637    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 11:34:26.847715 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:26.847698    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 11:34:26.847749 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:26.847715    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls podName:7b681f4b-5ee2-4693-8e41-4be4f0a374c1 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:42.847698862 +0000 UTC m=+67.047728642 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls") pod "dns-default-t27gs" (UID: "7b681f4b-5ee2-4693-8e41-4be4f0a374c1") : secret "dns-default-metrics-tls" not found
Sep 19 11:34:26.847806 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:26.847743    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-r6svd\" (UID: \"83c3531f-3ddd-45b9-91cd-3ac141ebd487\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-r6svd"
Sep 19 11:34:26.847806 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:26.847753    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls podName:e153e212-b7ce-4fea-89bd-970b816f0763 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:42.847738765 +0000 UTC m=+67.047768545 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-q9gbk" (UID: "e153e212-b7ce-4fea-89bd-970b816f0763") : secret "samples-operator-tls" not found
Sep 19 11:34:26.847806 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:26.847792    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 11:34:26.847959 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:26.847817    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert podName:83c3531f-3ddd-45b9-91cd-3ac141ebd487 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:42.847810007 +0000 UTC m=+67.047839790 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-r6svd" (UID: "83c3531f-3ddd-45b9-91cd-3ac141ebd487") : secret "networking-console-plugin-cert" not found
Sep 19 11:34:28.095812 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:28.095779    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:28.142513 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:28.142444    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:29.901332 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:29.901299    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:30.001827 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:30.001804    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:30.867944 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:30.867857    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:30.969664 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:30.966953    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:31.236016 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.235247    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm" event={"ID":"36391cd8-d035-4be4-bcb3-f33a076dd18a","Type":"ContainerStarted","Data":"7f7c35f3c5353f5f3e046aa56e7f02431d48129765577c37548fa3af82f1ee4e"}
Sep 19 11:34:31.248650 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.248064    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd" event={"ID":"cc7e845c-b6fd-4a96-8258-80b4e8f02e57","Type":"ContainerStarted","Data":"2ad883109fbf14373ac9fe9328dc64fc92ef0e4de4294373d9b37238d803e722"}
Sep 19 11:34:31.253261 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.252586    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f7cc664c4-w4qsm" podStartSLOduration=16.308337867 podStartE2EDuration="35.252566782s" totalImagesPullingTime="18.944228915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:34:31.251676301 +0000 UTC m=+55.451706104" watchObservedRunningTime="2026-09-19 11:34:31.252566782 +0000 UTC m=+55.452596584"
Sep 19 11:34:31.258932 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.258599    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-m9jqg" event={"ID":"96c0a054-f70d-46dd-b510-9056a47a6b9b","Type":"ContainerStarted","Data":"f1035eab25dc4bcc617f0870e9b6756e852dcde0e1c6c1bd5075b76bedb5df5a"}
Sep 19 11:34:31.269257 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.267695    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dpgkd" event={"ID":"f96896a2-5494-4bc6-81c9-154860ee5f6b","Type":"ContainerStarted","Data":"86efb79625b0a60c9977bcdda0a536d18f656ff60c60cc1b26ea7536312686b6"}
Sep 19 11:34:31.269662 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.269613    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd" podStartSLOduration=20.452693173 podStartE2EDuration="39.269592614s" totalImagesPullingTime="18.816899441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:34:31.268652909 +0000 UTC m=+55.468682712" watchObservedRunningTime="2026-09-19 11:34:31.269592614 +0000 UTC m=+55.469622419"
Sep 19 11:34:31.274023 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.273337    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hlvdj" event={"ID":"1936edc6-ecd0-46ca-9b8e-08cc8b2dc383","Type":"ContainerStarted","Data":"dd21557945a7c73bc2bd656f029e5b6447003c95e12bf13ee48fd54f92ed337e"}
Sep 19 11:34:31.276965 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.276657    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5774c959c7-6f759" event={"ID":"20620a5d-5b06-46ca-bff8-5322e24c32b2","Type":"ContainerStarted","Data":"0c66396a4072339b135bc8edb4121b77b72468b88d83616a983f5d7a82c956ee"}
Sep 19 11:34:31.277829 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.277660    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5774c959c7-6f759"
Sep 19 11:34:31.279886 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.279436    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h4xjv" event={"ID":"2066d78e-223c-4531-98aa-edc826b41c3b","Type":"ContainerStarted","Data":"63b54dda4d6fc9a68cc1831564a9234debaf5eab17e07a6eba5a1ad7c0db2a3b"}
Sep 19 11:34:31.281965 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.281860    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9zcpw" event={"ID":"40987fee-2f7a-4681-a32c-22aaa7b8a8f7","Type":"ContainerStarted","Data":"031d18fd140bca7b60d098f5c04b1fdffb1371be730e5e1bfc98bd17757f2a64"}
Sep 19 11:34:31.284386 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.284020    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m2dhj" event={"ID":"d0c272c9-c396-486e-b63b-6726cf1321d8","Type":"ContainerStarted","Data":"7b8e591e99dee3559d48e179ab1173ef52985aa879a0d538ffafc3b055186dee"}
Sep 19 11:34:31.284929 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.284757    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:34:31.290186 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.288773    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dpgkd" podStartSLOduration=34.736414187 podStartE2EDuration="53.288756781s" totalImagesPullingTime="18.552342594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 11:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:34:31.286755145 +0000 UTC m=+55.486784946" watchObservedRunningTime="2026-09-19 11:34:31.288756781 +0000 UTC m=+55.488786581"
Sep 19 11:34:31.290186 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.289151    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntx42" event={"ID":"8dd0a400-b99d-4d02-b42c-f05d4f40a484","Type":"ContainerStarted","Data":"c0ae7fcb6d686e4bf99f021cb43f80a67e997427c012e23a5379a614e88ed844"}
Sep 19 11:34:31.290186 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.290147    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5774c959c7-6f759"
Sep 19 11:34:31.292930 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.292469    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg" event={"ID":"c323067d-ee96-4a29-8322-83ce5f3f454f","Type":"ContainerStarted","Data":"6c4222abd2bff89662577f7057872c32d8dee9c1c23f2297baf48bc12e0f5219"}
Sep 19 11:34:31.292930 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.292501    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg" event={"ID":"c323067d-ee96-4a29-8322-83ce5f3f454f","Type":"ContainerStarted","Data":"08d7321b42904bd1b38f4ca56d8f3c104c5c37bfe2ca3f0858b49cb500a9ba4e"}
Sep 19 11:34:31.306656 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.306169    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ftf42" event={"ID":"aa857849-d4f4-4815-aa2d-8412e4a8033a","Type":"ContainerStarted","Data":"1641d488eaf60ff45ba80ad3082a4877965b3cb320ca4d5e469cad620540d1a7"}
Sep 19 11:34:31.314993 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.314268    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/0.log"
Sep 19 11:34:31.314993 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.314305    2584 generic.go:356] "Generic (PLEG): container finished" podID="24aae5c7-e112-496e-ba52-a52e02e520c1" containerID="4d2ab5046978718c7e275785024007cd1f29761d8f4897c083d71d64e60b1834" exitCode=255
Sep 19 11:34:31.314993 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.314359    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj" event={"ID":"24aae5c7-e112-496e-ba52-a52e02e520c1","Type":"ContainerDied","Data":"4d2ab5046978718c7e275785024007cd1f29761d8f4897c083d71d64e60b1834"}
Sep 19 11:34:31.314993 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.314665    2584 scope.go:117] "RemoveContainer" containerID="4d2ab5046978718c7e275785024007cd1f29761d8f4897c083d71d64e60b1834"
Sep 19 11:34:31.316087 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.315635    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-m9jqg" podStartSLOduration=20.974374708 podStartE2EDuration="39.315615136s" totalImagesPullingTime="18.341240428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:34:31.301455051 +0000 UTC m=+55.501484850" watchObservedRunningTime="2026-09-19 11:34:31.315615136 +0000 UTC m=+55.515644929"
Sep 19 11:34:31.322584 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.321182    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h4xjv" podStartSLOduration=3.091500297 podStartE2EDuration="21.321167419s" totalImagesPullingTime="18.229667122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:34:31.319893444 +0000 UTC m=+55.519923241" watchObservedRunningTime="2026-09-19 11:34:31.321167419 +0000 UTC m=+55.521197219"
Sep 19 11:34:31.384833 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.382878    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hlvdj" podStartSLOduration=20.537606913 podStartE2EDuration="39.382858405s" totalImagesPullingTime="18.845251492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:34:31.344058876 +0000 UTC m=+55.544088677" watchObservedRunningTime="2026-09-19 11:34:31.382858405 +0000 UTC m=+55.582888208"
Sep 19 11:34:31.384833 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.383200    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5774c959c7-6f759" podStartSLOduration=16.381988077 podStartE2EDuration="35.383191142s" totalImagesPullingTime="19.001203065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:34:31.381959425 +0000 UTC m=+55.581989225" watchObservedRunningTime="2026-09-19 11:34:31.383191142 +0000 UTC m=+55.583220944"
Sep 19 11:34:31.421614 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.421560    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9zcpw" podStartSLOduration=21.088807157 podStartE2EDuration="39.421539382s" totalImagesPullingTime="18.332732225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:34:31.403240052 +0000 UTC m=+55.603269856" watchObservedRunningTime="2026-09-19 11:34:31.421539382 +0000 UTC m=+55.621569184"
Sep 19 11:34:31.422235 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.422201    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m2dhj" podStartSLOduration=35.564820264 podStartE2EDuration="53.42218648s" totalImagesPullingTime="17.857366216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:34:31.420387579 +0000 UTC m=+55.620417381" watchObservedRunningTime="2026-09-19 11:34:31.42218648 +0000 UTC m=+55.622216285"
Sep 19 11:34:31.473724 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.473653    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ftf42" podStartSLOduration=20.466654796 podStartE2EDuration="39.473629665s" totalImagesPullingTime="19.006974869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:34:31.473481121 +0000 UTC m=+55.673510913" watchObservedRunningTime="2026-09-19 11:34:31.473629665 +0000 UTC m=+55.673659469"
Sep 19 11:34:31.487769 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.487669    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:31.538928 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.537893    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:31.943639 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:31.943550    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:32.041826 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.041791    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:32.321827 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.321796    2584 generic.go:356] "Generic (PLEG): container finished" podID="8dd0a400-b99d-4d02-b42c-f05d4f40a484" containerID="c0ae7fcb6d686e4bf99f021cb43f80a67e997427c012e23a5379a614e88ed844" exitCode=0
Sep 19 11:34:32.321827 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.321824    2584 generic.go:356] "Generic (PLEG): container finished" podID="8dd0a400-b99d-4d02-b42c-f05d4f40a484" containerID="3845fae5927a9ea5eaed8f5ce15bb0e1e55db4141e492ad8270ebe8becc208d0" exitCode=0
Sep 19 11:34:32.322071 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.321889    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntx42" event={"ID":"8dd0a400-b99d-4d02-b42c-f05d4f40a484","Type":"ContainerDied","Data":"c0ae7fcb6d686e4bf99f021cb43f80a67e997427c012e23a5379a614e88ed844"}
Sep 19 11:34:32.322071 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.321968    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntx42" event={"ID":"8dd0a400-b99d-4d02-b42c-f05d4f40a484","Type":"ContainerDied","Data":"3845fae5927a9ea5eaed8f5ce15bb0e1e55db4141e492ad8270ebe8becc208d0"}
Sep 19 11:34:32.324148 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.324125    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg" event={"ID":"c323067d-ee96-4a29-8322-83ce5f3f454f","Type":"ContainerStarted","Data":"84dc6ed33d66de0ef1157a4a69069f02ac98e7b870455273ff1cf5485b5f4a2f"}
Sep 19 11:34:32.325810 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.325787    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/1.log"
Sep 19 11:34:32.326296 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.326280    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/0.log"
Sep 19 11:34:32.326350 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.326320    2584 generic.go:356] "Generic (PLEG): container finished" podID="24aae5c7-e112-496e-ba52-a52e02e520c1" containerID="dba44067722fd6c765b5dca1fe54b38302739d4c9cbc7d9457262f6afa115dc8" exitCode=255
Sep 19 11:34:32.326440 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.326416    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj" event={"ID":"24aae5c7-e112-496e-ba52-a52e02e520c1","Type":"ContainerDied","Data":"dba44067722fd6c765b5dca1fe54b38302739d4c9cbc7d9457262f6afa115dc8"}
Sep 19 11:34:32.326545 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.326472    2584 scope.go:117] "RemoveContainer" containerID="4d2ab5046978718c7e275785024007cd1f29761d8f4897c083d71d64e60b1834"
Sep 19 11:34:32.326657 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.326634    2584 scope.go:117] "RemoveContainer" containerID="dba44067722fd6c765b5dca1fe54b38302739d4c9cbc7d9457262f6afa115dc8"
Sep 19 11:34:32.326829 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:32.326808    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-4bxfj_openshift-console-operator(24aae5c7-e112-496e-ba52-a52e02e520c1)\"" pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj" podUID="24aae5c7-e112-496e-ba52-a52e02e520c1"
Sep 19 11:34:32.384451 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.384391    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg" podStartSLOduration=17.512320079 podStartE2EDuration="36.384372317s" totalImagesPullingTime="18.872052238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:34:32.38364831 +0000 UTC m=+56.583678111" watchObservedRunningTime="2026-09-19 11:34:32.384372317 +0000 UTC m=+56.584402118"
Sep 19 11:34:32.467312 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.467278    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-t2fzm"]
Sep 19 11:34:32.469770 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.469750    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:32.472482 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.472458    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 19 11:34:32.472482 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.472477    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g979l\""
Sep 19 11:34:32.472689 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.472574    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 19 11:34:32.482694 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.482651    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:32.482812 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.482787    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t2fzm"]
Sep 19 11:34:32.508692 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.508660    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/047fafca-cf10-4883-aad0-0c68166d05e0-data-volume\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:32.508843 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.508829    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/047fafca-cf10-4883-aad0-0c68166d05e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:32.508885 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.508853    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n677\" (UniqueName: \"kubernetes.io/projected/047fafca-cf10-4883-aad0-0c68166d05e0-kube-api-access-9n677\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:32.508930 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.508891    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/047fafca-cf10-4883-aad0-0c68166d05e0-crio-socket\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:32.509003 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.508977    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:32.547132 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.547085    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:32.610030 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.609958    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/047fafca-cf10-4883-aad0-0c68166d05e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:32.610030 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.609988    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n677\" (UniqueName: \"kubernetes.io/projected/047fafca-cf10-4883-aad0-0c68166d05e0-kube-api-access-9n677\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:32.610247 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.610034    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/047fafca-cf10-4883-aad0-0c68166d05e0-crio-socket\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:32.610247 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.610060    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:32.610247 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:32.610078    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 11:34:32.610247 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.610087    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/047fafca-cf10-4883-aad0-0c68166d05e0-data-volume\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:32.610247 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:32.610155    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/047fafca-cf10-4883-aad0-0c68166d05e0-kube-rbac-proxy-cm podName:047fafca-cf10-4883-aad0-0c68166d05e0 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:33.110135698 +0000 UTC m=+57.310165498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/047fafca-cf10-4883-aad0-0c68166d05e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t2fzm" (UID: "047fafca-cf10-4883-aad0-0c68166d05e0") : configmap "kube-rbac-proxy" not found
Sep 19 11:34:32.610247 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.610160    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/047fafca-cf10-4883-aad0-0c68166d05e0-crio-socket\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:32.610247 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:32.610198    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 11:34:32.610496 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:32.610259    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls podName:047fafca-cf10-4883-aad0-0c68166d05e0 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:33.110245234 +0000 UTC m=+57.310275025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t2fzm" (UID: "047fafca-cf10-4883-aad0-0c68166d05e0") : secret "insights-runtime-extractor-tls" not found
Sep 19 11:34:32.610496 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.610375    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/047fafca-cf10-4883-aad0-0c68166d05e0-data-volume\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:32.624685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.624650    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n677\" (UniqueName: \"kubernetes.io/projected/047fafca-cf10-4883-aad0-0c68166d05e0-kube-api-access-9n677\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:32.645676 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.645650    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:32.747165 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.747097    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:32.803828 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:32.803792    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:33.113765 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:33.113713    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/047fafca-cf10-4883-aad0-0c68166d05e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:33.113765 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:33.113783    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:33.114260 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:33.113839    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 11:34:33.114260 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:33.113905    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/047fafca-cf10-4883-aad0-0c68166d05e0-kube-rbac-proxy-cm podName:047fafca-cf10-4883-aad0-0c68166d05e0 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:34.11388989 +0000 UTC m=+58.313919670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/047fafca-cf10-4883-aad0-0c68166d05e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t2fzm" (UID: "047fafca-cf10-4883-aad0-0c68166d05e0") : configmap "kube-rbac-proxy" not found
Sep 19 11:34:33.114260 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:33.113934    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 11:34:33.114260 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:33.113984    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls podName:047fafca-cf10-4883-aad0-0c68166d05e0 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:34.113973936 +0000 UTC m=+58.314003716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t2fzm" (UID: "047fafca-cf10-4883-aad0-0c68166d05e0") : secret "insights-runtime-extractor-tls" not found
Sep 19 11:34:33.334456 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:33.334414    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntx42" event={"ID":"8dd0a400-b99d-4d02-b42c-f05d4f40a484","Type":"ContainerStarted","Data":"357c28062dd03ca24b8dd861991c1b70e153b69e182619e681b3e32af97fa542"}
Sep 19 11:34:33.336069 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:33.336050    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/1.log"
Sep 19 11:34:33.336428 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:33.336411    2584 scope.go:117] "RemoveContainer" containerID="dba44067722fd6c765b5dca1fe54b38302739d4c9cbc7d9457262f6afa115dc8"
Sep 19 11:34:33.336611 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:33.336591    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-4bxfj_openshift-console-operator(24aae5c7-e112-496e-ba52-a52e02e520c1)\"" pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj" podUID="24aae5c7-e112-496e-ba52-a52e02e520c1"
Sep 19 11:34:33.360280 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:33.360216    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ntx42" podStartSLOduration=10.454747221 podStartE2EDuration="55.36019393s" totalImagesPullingTime="44.905446709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 11:33:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:34:33.356483758 +0000 UTC m=+57.556513560" watchObservedRunningTime="2026-09-19 11:34:33.36019393 +0000 UTC m=+57.560223735"
Sep 19 11:34:33.441209 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:33.441103    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:33.491708 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:33.491665    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:33.540624 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:33.540554    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:34.123929 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:34.123888    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/047fafca-cf10-4883-aad0-0c68166d05e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:34.124407 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:34.123948    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:34.124407 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:34.124032    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 11:34:34.124407 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:34.124068    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 11:34:34.124407 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:34.124115    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls podName:047fafca-cf10-4883-aad0-0c68166d05e0 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:36.12409921 +0000 UTC m=+60.324128994 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t2fzm" (UID: "047fafca-cf10-4883-aad0-0c68166d05e0") : secret "insights-runtime-extractor-tls" not found
Sep 19 11:34:34.124407 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:34.124129    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/047fafca-cf10-4883-aad0-0c68166d05e0-kube-rbac-proxy-cm podName:047fafca-cf10-4883-aad0-0c68166d05e0 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:36.1241222 +0000 UTC m=+60.324151980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/047fafca-cf10-4883-aad0-0c68166d05e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t2fzm" (UID: "047fafca-cf10-4883-aad0-0c68166d05e0") : configmap "kube-rbac-proxy" not found
Sep 19 11:34:34.243567 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:34.243540    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vc5vt_869b7a50-52b6-4167-aa2d-f5152f2c8eb4/dns-node-resolver/0.log"
Sep 19 11:34:34.273674 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:34.273642    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:34.373645 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:34.373604    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:35.111111 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:35.111079    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:35.244339 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:35.244309    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rsnfw_b2e7da35-ed65-4cd7-8d44-9b25ff53fba2/node-ca/0.log"
Sep 19 11:34:35.883236 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:35.883206    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h4xjv_2066d78e-223c-4531-98aa-edc826b41c3b/serve-healthcheck-canary/0.log"
Sep 19 11:34:36.144230 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:36.144137    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/047fafca-cf10-4883-aad0-0c68166d05e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:36.144230 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:36.144196    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:36.144416 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:36.144265    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 11:34:36.144416 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:36.144293    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 11:34:36.144416 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:36.144335    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/047fafca-cf10-4883-aad0-0c68166d05e0-kube-rbac-proxy-cm podName:047fafca-cf10-4883-aad0-0c68166d05e0 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:40.14431649 +0000 UTC m=+64.344346271 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/047fafca-cf10-4883-aad0-0c68166d05e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t2fzm" (UID: "047fafca-cf10-4883-aad0-0c68166d05e0") : configmap "kube-rbac-proxy" not found
Sep 19 11:34:36.144416 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:36.144352    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls podName:047fafca-cf10-4883-aad0-0c68166d05e0 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:40.144344538 +0000 UTC m=+64.344374318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t2fzm" (UID: "047fafca-cf10-4883-aad0-0c68166d05e0") : secret "insights-runtime-extractor-tls" not found
Sep 19 11:34:37.533704 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:37.533663    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:39.055050 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:39.055016    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:39.164180 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:39.164141    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:39.267267 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:39.267235    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:39.366736 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:39.366698    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:39.446340 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:39.446308    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:40.151837 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:40.151803    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:40.179046 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:40.179010    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/047fafca-cf10-4883-aad0-0c68166d05e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:40.179141 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:40.179064    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:40.179141 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:40.179131    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 11:34:40.179215 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:40.179163    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 11:34:40.179215 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:40.179208    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/047fafca-cf10-4883-aad0-0c68166d05e0-kube-rbac-proxy-cm podName:047fafca-cf10-4883-aad0-0c68166d05e0 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:48.179191011 +0000 UTC m=+72.379220814 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/047fafca-cf10-4883-aad0-0c68166d05e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t2fzm" (UID: "047fafca-cf10-4883-aad0-0c68166d05e0") : configmap "kube-rbac-proxy" not found
Sep 19 11:34:40.179296 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:40.179222    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls podName:047fafca-cf10-4883-aad0-0c68166d05e0 nodeName:}" failed. No retries permitted until 2026-09-19 11:34:48.17921629 +0000 UTC m=+72.379246071 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t2fzm" (UID: "047fafca-cf10-4883-aad0-0c68166d05e0") : secret "insights-runtime-extractor-tls" not found
Sep 19 11:34:40.755581 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:40.755538    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:40.953716 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:40.953683    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:41.162447 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:41.162407    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:41.245407 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:41.245364    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj"
Sep 19 11:34:41.245610 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:41.245480    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj"
Sep 19 11:34:41.245905 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:41.245888    2584 scope.go:117] "RemoveContainer" containerID="dba44067722fd6c765b5dca1fe54b38302739d4c9cbc7d9457262f6afa115dc8"
Sep 19 11:34:41.246128 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:41.246104    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-4bxfj_openshift-console-operator(24aae5c7-e112-496e-ba52-a52e02e520c1)\"" pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj" podUID="24aae5c7-e112-496e-ba52-a52e02e520c1"
Sep 19 11:34:41.554763 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:41.554584    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:41.896898 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:41.896853    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:42.381134 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:42.381101    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:42.558513 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:42.558485    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:42.802250 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:42.802153    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:42.802250 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:42.802227    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:34:42.802475 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:42.802261    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:42.802475 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:42.802280    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nll25\" (UID: \"53ad775e-63df-4c3d-856d-f17723aed41e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"
Sep 19 11:34:42.802475 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:42.802318    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle podName:5b388e80-43fb-4367-a7c1-6c53f04caffa nodeName:}" failed. No retries permitted until 2026-09-19 11:35:14.8022959 +0000 UTC m=+99.002325700 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle") pod "router-default-687866bb-9xvr5" (UID: "5b388e80-43fb-4367-a7c1-6c53f04caffa") : configmap references non-existent config key: service-ca.crt
Sep 19 11:34:42.802475 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:42.802364    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 11:34:42.802475 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:42.802368    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 11:34:42.802475 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:42.802411    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls podName:53ad775e-63df-4c3d-856d-f17723aed41e nodeName:}" failed. No retries permitted until 2026-09-19 11:35:14.802400607 +0000 UTC m=+99.002430387 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-nll25" (UID: "53ad775e-63df-4c3d-856d-f17723aed41e") : secret "cluster-monitoring-operator-tls" not found
Sep 19 11:34:42.802475 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:42.802425    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs podName:5b388e80-43fb-4367-a7c1-6c53f04caffa nodeName:}" failed. No retries permitted until 2026-09-19 11:35:14.802417661 +0000 UTC m=+99.002447441 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs") pod "router-default-687866bb-9xvr5" (UID: "5b388e80-43fb-4367-a7c1-6c53f04caffa") : secret "router-metrics-certs-default" not found
Sep 19 11:34:42.804883 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:42.804859    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls\") pod \"image-registry-657b49657d-km2v8\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") " pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:42.903102 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:42.903061    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-r6svd\" (UID: \"83c3531f-3ddd-45b9-91cd-3ac141ebd487\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-r6svd"
Sep 19 11:34:42.903306 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:42.903160    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls\") pod \"dns-default-t27gs\" (UID: \"7b681f4b-5ee2-4693-8e41-4be4f0a374c1\") " pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:42.903306 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:42.903188    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-q9gbk\" (UID: \"e153e212-b7ce-4fea-89bd-970b816f0763\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk"
Sep 19 11:34:42.903306 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:42.903208    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 11:34:42.903306 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:42.903283    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert podName:83c3531f-3ddd-45b9-91cd-3ac141ebd487 nodeName:}" failed. No retries permitted until 2026-09-19 11:35:14.903265951 +0000 UTC m=+99.103295732 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-r6svd" (UID: "83c3531f-3ddd-45b9-91cd-3ac141ebd487") : secret "networking-console-plugin-cert" not found
Sep 19 11:34:42.905681 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:42.905654    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b681f4b-5ee2-4693-8e41-4be4f0a374c1-metrics-tls\") pod \"dns-default-t27gs\" (UID: \"7b681f4b-5ee2-4693-8e41-4be4f0a374c1\") " pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:42.905830 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:42.905813    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e153e212-b7ce-4fea-89bd-970b816f0763-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-q9gbk\" (UID: \"e153e212-b7ce-4fea-89bd-970b816f0763\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk"
Sep 19 11:34:42.950328 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:42.950292    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:43.005489 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:43.005455    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:43.073354 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:43.073326    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sbk8r\""
Sep 19 11:34:43.081535 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:43.081485    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:43.098234 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:43.098202    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mvj2p\""
Sep 19 11:34:43.104966 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:43.104944    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5jm8q\""
Sep 19 11:34:43.105974 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:43.105955    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:43.113233 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:43.113208    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk"
Sep 19 11:34:43.159067 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:43.158878    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:43.361274 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:43.361195    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-657b49657d-km2v8"]
Sep 19 11:34:43.364496 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:34:43.364471    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60111092_bf1d_4949_b3a2_fc36e4a01d59.slice/crio-11723254c09d25254fa7f0b08d8761cef95624daca602241443ea1e7b1dd795f WatchSource:0}: Error finding container 11723254c09d25254fa7f0b08d8761cef95624daca602241443ea1e7b1dd795f: Status 404 returned error can't find the container with id 11723254c09d25254fa7f0b08d8761cef95624daca602241443ea1e7b1dd795f
Sep 19 11:34:43.371348 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:43.371305    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657b49657d-km2v8" event={"ID":"60111092-bf1d-4949-b3a2-fc36e4a01d59","Type":"ContainerStarted","Data":"11723254c09d25254fa7f0b08d8761cef95624daca602241443ea1e7b1dd795f"}
Sep 19 11:34:43.392852 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:43.392825    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t27gs"]
Sep 19 11:34:43.397329 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:34:43.397292    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b681f4b_5ee2_4693_8e41_4be4f0a374c1.slice/crio-b31b5c70c6164f15e58a31992005b96ff652b02418f238148271881518e84187 WatchSource:0}: Error finding container b31b5c70c6164f15e58a31992005b96ff652b02418f238148271881518e84187: Status 404 returned error can't find the container with id b31b5c70c6164f15e58a31992005b96ff652b02418f238148271881518e84187
Sep 19 11:34:43.398393 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:43.398369    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk"]
Sep 19 11:34:43.609968 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:43.609931    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs\") pod \"network-metrics-daemon-pgph6\" (UID: \"83707003-2551-42b5-8943-8243dd6daa80\") " pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:34:43.612421 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:43.612359    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 11:34:43.620671 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:43.620644    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 19 11:34:43.620805 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:43.620735    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs podName:83707003-2551-42b5-8943-8243dd6daa80 nodeName:}" failed. No retries permitted until 2026-09-19 11:35:47.620712142 +0000 UTC m=+131.820741929 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs") pod "network-metrics-daemon-pgph6" (UID: "83707003-2551-42b5-8943-8243dd6daa80") : secret "metrics-daemon-secret" not found
Sep 19 11:34:43.757262 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:43.757223    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:43.819632 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:43.819604    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:44.157403 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:44.157372    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:44.377314 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:44.377274    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657b49657d-km2v8" event={"ID":"60111092-bf1d-4949-b3a2-fc36e4a01d59","Type":"ContainerStarted","Data":"a17455a0d65c27cb2b58b268d51c75392641df12a5f91170aa0b17560843ee4b"}
Sep 19 11:34:44.377506 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:44.377384    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:34:44.379210 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:44.379178    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk" event={"ID":"e153e212-b7ce-4fea-89bd-970b816f0763","Type":"ContainerStarted","Data":"9d7b84c9dc0931fdc786c566d891d347fe3db0cb8f48fd77f5bc43318adf3fae"}
Sep 19 11:34:44.380738 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:44.380710    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t27gs" event={"ID":"7b681f4b-5ee2-4693-8e41-4be4f0a374c1","Type":"ContainerStarted","Data":"b31b5c70c6164f15e58a31992005b96ff652b02418f238148271881518e84187"}
Sep 19 11:34:44.398221 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:44.398159    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-657b49657d-km2v8" podStartSLOduration=66.39814205 podStartE2EDuration="1m6.39814205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:34:44.39622129 +0000 UTC m=+68.596251094" watchObservedRunningTime="2026-09-19 11:34:44.39814205 +0000 UTC m=+68.598171851"
Sep 19 11:34:44.555543 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:44.555459    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:44.656155 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:44.656113    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:44.757294 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:44.757213    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:44.958577 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:44.958507    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:45.134687 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:45.134656    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:45.352907 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:45.352874    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:45.524850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:45.524777    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:45.576262 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:45.576226    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:46.153296 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:46.153259    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:46.352891 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:46.352852    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:46.390873 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:46.390839    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t27gs" event={"ID":"7b681f4b-5ee2-4693-8e41-4be4f0a374c1","Type":"ContainerStarted","Data":"f89ccc26f4fdbfab61c53945a186635b5832c7e114dadb5694f927c87b33caae"}
Sep 19 11:34:46.391078 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:46.390896    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t27gs" event={"ID":"7b681f4b-5ee2-4693-8e41-4be4f0a374c1","Type":"ContainerStarted","Data":"4eb4fa6c390c6d7f8b6a8afd121f531601ae31b982a91a1662482e44d99cb52a"}
Sep 19 11:34:46.391078 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:46.390947    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:46.392731 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:46.392707    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk" event={"ID":"e153e212-b7ce-4fea-89bd-970b816f0763","Type":"ContainerStarted","Data":"846306f2f75eb01bb47f9837abaeed886475021583499a4c5e97757ec683885a"}
Sep 19 11:34:46.392857 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:46.392754    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk" event={"ID":"e153e212-b7ce-4fea-89bd-970b816f0763","Type":"ContainerStarted","Data":"545586e3598ec8dff2d8741ab04874770dd02b932e633df3cbe0ec1041a6c6ca"}
Sep 19 11:34:46.407258 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:46.407144    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-t27gs" podStartSLOduration=34.199574854 podStartE2EDuration="36.40712762s" totalImagesPullingTime="2.207552766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:34:46.406050566 +0000 UTC m=+70.606080369" watchObservedRunningTime="2026-09-19 11:34:46.40712762 +0000 UTC m=+70.607157422"
Sep 19 11:34:46.421582 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:46.421522    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-q9gbk" podStartSLOduration=52.258397613 podStartE2EDuration="54.42150305s" totalImagesPullingTime="2.163105437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:34:46.420145527 +0000 UTC m=+70.620175329" watchObservedRunningTime="2026-09-19 11:34:46.42150305 +0000 UTC m=+70.621532911"
Sep 19 11:34:46.751884 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:46.751781    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:47.350144 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:47.350112    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:47.755864 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:47.755671    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:48.153284 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:48.153245    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:48.256468 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:48.256422    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/047fafca-cf10-4883-aad0-0c68166d05e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:48.256653 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:48.256494    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:48.256697 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:48.256648    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 11:34:48.256735 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:34:48.256718    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls podName:047fafca-cf10-4883-aad0-0c68166d05e0 nodeName:}" failed. No retries permitted until 2026-09-19 11:35:04.256701819 +0000 UTC m=+88.456731600 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t2fzm" (UID: "047fafca-cf10-4883-aad0-0c68166d05e0") : secret "insights-runtime-extractor-tls" not found
Sep 19 11:34:48.257032 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:48.257012    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/047fafca-cf10-4883-aad0-0c68166d05e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:34:48.354764 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:48.354722    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:48.953558 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:48.953524    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:49.353126 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:49.353093    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:49.555675 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:49.555641    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:49.755161 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:49.755058    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:52.224161 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:52.224133    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:52.291897 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:52.291868    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:52.391868 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:52.391836    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:52.491544 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:52.491452    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:52.542584 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:52.542552    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:52.897270 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:52.897241    2584 scope.go:117] "RemoveContainer" containerID="dba44067722fd6c765b5dca1fe54b38302739d4c9cbc7d9457262f6afa115dc8"
Sep 19 11:34:53.420531 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:53.420506    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/1.log"
Sep 19 11:34:53.420983 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:53.420575    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj" event={"ID":"24aae5c7-e112-496e-ba52-a52e02e520c1","Type":"ContainerStarted","Data":"b39fa98263ed7b0e1d7889adaee608bbd1960b8c084df36cab18148a4e80fcaa"}
Sep 19 11:34:53.420983 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:53.420950    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj"
Sep 19 11:34:53.438936 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:53.438871    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj" podStartSLOduration=42.425381277 podStartE2EDuration="1m1.4388536s" totalImagesPullingTime="19.013472323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:34:53.43827253 +0000 UTC m=+77.638302345" watchObservedRunningTime="2026-09-19 11:34:53.4388536 +0000 UTC m=+77.638883401"
Sep 19 11:34:53.481629 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:53.481599    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-4bxfj"
Sep 19 11:34:53.661099 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:53.661053    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:53.726784 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:53.726698    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:53.781041 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:53.781001    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:53.951683 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:53.951641    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:54.129182 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:54.129144    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:54.528995 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:54.528893    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:54.582359 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:54.582330    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:54.731541 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:54.731500    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:54.831482 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:54.831455    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:55.347013 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:55.346984    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:55.540671 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:55.540643    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:55.840938 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:55.840880    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:56.398612 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:56.398581    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t27gs"
Sep 19 11:34:56.760255 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:56.760171    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:56.811066 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:56.811028    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:58.114938 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:58.114882    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:58.164803 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:58.164776    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:58.468482 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:58.468397    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:58.539191 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:58.539149    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:34:58.620089 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:34:58.620052    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:02.329384 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:02.329354    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m2dhj"
Sep 19 11:35:02.454570 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:02.454539    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:02.504713 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:02.504675    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:03.085722 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:03.085684    2584 patch_prober.go:28] interesting pod/image-registry-657b49657d-km2v8 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 19 11:35:03.085885 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:03.085742    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-657b49657d-km2v8" podUID="60111092-bf1d-4949-b3a2-fc36e4a01d59" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 11:35:04.308540 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:04.308496    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:35:04.311146 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:04.311114    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/047fafca-cf10-4883-aad0-0c68166d05e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t2fzm\" (UID: \"047fafca-cf10-4883-aad0-0c68166d05e0\") " pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:35:04.581683 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:04.581652    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g979l\""
Sep 19 11:35:04.589770 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:04.589741    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t2fzm"
Sep 19 11:35:04.904726 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:04.904702    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t2fzm"]
Sep 19 11:35:04.907095 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:35:04.907063    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047fafca_cf10_4883_aad0_0c68166d05e0.slice/crio-419867bd5f3d3a826d561db02c06e9ad70ffbe6cde5eb7d2c320621842c1cd37 WatchSource:0}: Error finding container 419867bd5f3d3a826d561db02c06e9ad70ffbe6cde5eb7d2c320621842c1cd37: Status 404 returned error can't find the container with id 419867bd5f3d3a826d561db02c06e9ad70ffbe6cde5eb7d2c320621842c1cd37
Sep 19 11:35:05.389809 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:05.389782    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:35:05.465210 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:05.465179    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t2fzm" event={"ID":"047fafca-cf10-4883-aad0-0c68166d05e0","Type":"ContainerStarted","Data":"88b1a121d526c39cc59b01c6649889cab5c6815d77a7a5bf9be06b12c20c53e9"}
Sep 19 11:35:05.465359 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:05.465237    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t2fzm" event={"ID":"047fafca-cf10-4883-aad0-0c68166d05e0","Type":"ContainerStarted","Data":"419867bd5f3d3a826d561db02c06e9ad70ffbe6cde5eb7d2c320621842c1cd37"}
Sep 19 11:35:05.509182 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:05.509143    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:05.560522 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:05.560494    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:05.645574 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:05.645409    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:05.695258 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:05.695229    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:06.358712 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:06.358679    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:06.411137 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:06.411104    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:06.471223 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:06.471184    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t2fzm" event={"ID":"047fafca-cf10-4883-aad0-0c68166d05e0","Type":"ContainerStarted","Data":"519a9c50d9bbbcc163382acf44f8fe4f7f8946abd1d6511671e691a3e947eb01"}
Sep 19 11:35:07.476957 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:07.476836    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t2fzm" event={"ID":"047fafca-cf10-4883-aad0-0c68166d05e0","Type":"ContainerStarted","Data":"4d20f51b07933e4c790d56b9f518753841970bdc0105630fea5da4618bd0338f"}
Sep 19 11:35:07.494252 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:07.494198    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-t2fzm" podStartSLOduration=33.426595377 podStartE2EDuration="35.494180911s" totalImagesPullingTime="2.067585534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 11:34:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:35:07.492497731 +0000 UTC m=+91.692527534" watchObservedRunningTime="2026-09-19 11:35:07.494180911 +0000 UTC m=+91.694210737"
Sep 19 11:35:07.886689 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:07.886656    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:07.987789 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:07.987755    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:08.093581 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:08.093546    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:09.019502 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:09.019468    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:09.069143 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:09.069103    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:09.363504 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:09.363471    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:09.420327 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:09.420296    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:10.962879 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:10.962830    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:11.063643 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:11.063608    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:13.454942 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:13.454899    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:13.556249 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:13.556219    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:14.007268 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:14.007214    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-657b49657d-km2v8"]
Sep 19 11:35:14.097143 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:14.097109    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:14.196705 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:14.196664    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:14.297547 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:14.297462    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:14.898605 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:14.898572    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:35:14.899024 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:14.898623    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:35:14.899024 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:14.898656    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nll25\" (UID: \"53ad775e-63df-4c3d-856d-f17723aed41e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"
Sep 19 11:35:14.899284 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:14.899259    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b388e80-43fb-4367-a7c1-6c53f04caffa-service-ca-bundle\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:35:14.901209 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:14.901183    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/53ad775e-63df-4c3d-856d-f17723aed41e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nll25\" (UID: \"53ad775e-63df-4c3d-856d-f17723aed41e\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"
Sep 19 11:35:14.901317 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:14.901288    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b388e80-43fb-4367-a7c1-6c53f04caffa-metrics-certs\") pod \"router-default-687866bb-9xvr5\" (UID: \"5b388e80-43fb-4367-a7c1-6c53f04caffa\") " pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:35:14.999146 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:14.999106    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-r6svd\" (UID: \"83c3531f-3ddd-45b9-91cd-3ac141ebd487\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-r6svd"
Sep 19 11:35:15.001779 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:15.001741    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c3531f-3ddd-45b9-91cd-3ac141ebd487-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-r6svd\" (UID: \"83c3531f-3ddd-45b9-91cd-3ac141ebd487\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-r6svd"
Sep 19 11:35:15.154482 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:15.154395    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4gb6j\""
Sep 19 11:35:15.161837 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:15.161813    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:35:15.164507 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:15.164312    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-j2npf\""
Sep 19 11:35:15.172475 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:15.172448    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"
Sep 19 11:35:15.241328 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:15.241300    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xhcrd\""
Sep 19 11:35:15.249259 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:15.249227    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-r6svd"
Sep 19 11:35:15.440710 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:15.440680    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-687866bb-9xvr5"]
Sep 19 11:35:15.443704 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:35:15.443665    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b388e80_43fb_4367_a7c1_6c53f04caffa.slice/crio-dee4847fc2d376196c0083a2faf11af61d3cc976b8dfc51985a8ebe962ebeb51 WatchSource:0}: Error finding container dee4847fc2d376196c0083a2faf11af61d3cc976b8dfc51985a8ebe962ebeb51: Status 404 returned error can't find the container with id dee4847fc2d376196c0083a2faf11af61d3cc976b8dfc51985a8ebe962ebeb51
Sep 19 11:35:15.444958 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:15.444932    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25"]
Sep 19 11:35:15.448006 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:35:15.447973    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ad775e_63df_4c3d_856d_f17723aed41e.slice/crio-f5bc3f51a4fd70434018a49af138493c58eb941c8d65f15b6ee4fa5fca0b60c8 WatchSource:0}: Error finding container f5bc3f51a4fd70434018a49af138493c58eb941c8d65f15b6ee4fa5fca0b60c8: Status 404 returned error can't find the container with id f5bc3f51a4fd70434018a49af138493c58eb941c8d65f15b6ee4fa5fca0b60c8
Sep 19 11:35:15.509078 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:15.509046    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25" event={"ID":"53ad775e-63df-4c3d-856d-f17723aed41e","Type":"ContainerStarted","Data":"f5bc3f51a4fd70434018a49af138493c58eb941c8d65f15b6ee4fa5fca0b60c8"}
Sep 19 11:35:15.510986 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:15.510956    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-687866bb-9xvr5" event={"ID":"5b388e80-43fb-4367-a7c1-6c53f04caffa","Type":"ContainerStarted","Data":"dee4847fc2d376196c0083a2faf11af61d3cc976b8dfc51985a8ebe962ebeb51"}
Sep 19 11:35:15.522563 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:15.522533    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-r6svd"]
Sep 19 11:35:15.525336 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:35:15.525295    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c3531f_3ddd_45b9_91cd_3ac141ebd487.slice/crio-e18f60250ef9d0da117c9c0c45b49719e0c68bebecbe0883132c813b074bbbe2 WatchSource:0}: Error finding container e18f60250ef9d0da117c9c0c45b49719e0c68bebecbe0883132c813b074bbbe2: Status 404 returned error can't find the container with id e18f60250ef9d0da117c9c0c45b49719e0c68bebecbe0883132c813b074bbbe2
Sep 19 11:35:15.739679 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:15.739592    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:15.842566 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:15.842521    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:15.941179 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:15.941141    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:16.517932 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:16.517873    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-687866bb-9xvr5" event={"ID":"5b388e80-43fb-4367-a7c1-6c53f04caffa","Type":"ContainerStarted","Data":"6b56e11d2b397c9487254d24288e4111e5b40b43b613e1c2b4bcad5edbae1ab6"}
Sep 19 11:35:16.519533 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:16.519500    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-r6svd" event={"ID":"83c3531f-3ddd-45b9-91cd-3ac141ebd487","Type":"ContainerStarted","Data":"e18f60250ef9d0da117c9c0c45b49719e0c68bebecbe0883132c813b074bbbe2"}
Sep 19 11:35:16.537828 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:16.537753    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-687866bb-9xvr5" podStartSLOduration=84.537732497 podStartE2EDuration="1m24.537732497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:35:16.53639754 +0000 UTC m=+100.736427343" watchObservedRunningTime="2026-09-19 11:35:16.537732497 +0000 UTC m=+100.737762300"
Sep 19 11:35:16.732426 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:16.732376    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:17.046857 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:17.046799    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:17.151536 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:17.151493    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:17.162239 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:17.162198    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:35:17.165807 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:17.165773    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:35:17.210389 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:17.210352    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:17.525328 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:17.525287    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25" event={"ID":"53ad775e-63df-4c3d-856d-f17723aed41e","Type":"ContainerStarted","Data":"ed369aa7a579e48d15e16476e767f1635966b1eb719b21aa72bce3e988f63d77"}
Sep 19 11:35:17.527177 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:17.527145    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-r6svd" event={"ID":"83c3531f-3ddd-45b9-91cd-3ac141ebd487","Type":"ContainerStarted","Data":"ce9c69bbe09ea1cd4eb60821d9d5d84c83b5e70fafa32fb4f51cd6e174dc1189"}
Sep 19 11:35:17.527437 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:17.527418    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:35:17.528748 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:17.528719    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-687866bb-9xvr5"
Sep 19 11:35:17.541815 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:17.541751    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nll25" podStartSLOduration=83.635512325 podStartE2EDuration="1m25.541731873s" totalImagesPullingTime="1.906219548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:35:17.54121665 +0000 UTC m=+101.741246453" watchObservedRunningTime="2026-09-19 11:35:17.541731873 +0000 UTC m=+101.741761675"
Sep 19 11:35:17.584337 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:17.582358    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-r6svd" podStartSLOduration=73.756974941 podStartE2EDuration="1m15.58233693s" totalImagesPullingTime="1.825361989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:35:17.581764549 +0000 UTC m=+101.781794354" watchObservedRunningTime="2026-09-19 11:35:17.58233693 +0000 UTC m=+101.782366731"
Sep 19 11:35:17.677539 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:17.677411    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:17.779155 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:17.779123    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:17.864315 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:17.864281    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:17.921134 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:17.921102    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:18.052054 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:18.051965    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:18.154281 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:18.154231    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:18.251895 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:18.251860    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:19.151718 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:19.151685    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:19.205392 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:19.205354    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:23.091095 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:23.091063    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:23.145140 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:23.145110    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:23.893637 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:23.893608    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:23.958311 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:23.958280    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:24.273977 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:24.273889    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:26.114680 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:26.114643    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:26.170106 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:26.170072    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:26.668751 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:26.668716    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:28.454887 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:28.454856    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:28.515709 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:28.515677    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:29.028038 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.028007    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:29.127952 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.127877    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:29.185872 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.185838    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:29.264266 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.264231    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:29.277930 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.277882    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fv299"]
Sep 19 11:35:29.281001 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.280940    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.283622 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.283589    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 19 11:35:29.283782 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.283717    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5clwj\""
Sep 19 11:35:29.283782 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.283760    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 19 11:35:29.284864 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.284846    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 19 11:35:29.284959 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.284847    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 19 11:35:29.358757 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.358722    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:29.429674 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.429640    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-sys\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.429864 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.429691    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-node-exporter-textfile\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.429864 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.429714    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.429864 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.429731    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-root\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.429864 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.429799    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-node-exporter-wtmp\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.429864 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.429850    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jqp5\" (UniqueName: \"kubernetes.io/projected/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-kube-api-access-4jqp5\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.430068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.429877    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-metrics-client-ca\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.430068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.429896    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-node-exporter-accelerators-collector-config\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.430068 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.429936    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-node-exporter-tls\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.458385 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.458350    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:29.530685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.530649    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-node-exporter-wtmp\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.530685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.530691    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jqp5\" (UniqueName: \"kubernetes.io/projected/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-kube-api-access-4jqp5\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.530890 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.530824    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-metrics-client-ca\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.530890 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.530863    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-node-exporter-accelerators-collector-config\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.530890 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.530833    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-node-exporter-wtmp\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.531071 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.530892    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-node-exporter-tls\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.531071 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.531013    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-sys\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.531071 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:35:29.531037    2584 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 19 11:35:29.531071 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.531050    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-node-exporter-textfile\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.531071 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.531067    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.531309 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.531091    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-root\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.531309 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.531121    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-sys\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.531309 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:35:29.531139    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-node-exporter-tls podName:4c7a5a73-a310-4974-9fb9-58220b1ae2c8 nodeName:}" failed. No retries permitted until 2026-09-19 11:35:30.031116634 +0000 UTC m=+114.231146419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-node-exporter-tls") pod "node-exporter-fv299" (UID: "4c7a5a73-a310-4974-9fb9-58220b1ae2c8") : secret "node-exporter-tls" not found
Sep 19 11:35:29.531309 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.531178    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-root\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.531511 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.531478    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-node-exporter-textfile\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.531511 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.531479    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-metrics-client-ca\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.531738 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.531717    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-node-exporter-accelerators-collector-config\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.533683 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.533662    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.539776 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.539748    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jqp5\" (UniqueName: \"kubernetes.io/projected/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-kube-api-access-4jqp5\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:29.559051 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.559005    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:29.659363 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.659328    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:29.760477 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.760436    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:29.845343 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:29.845306    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:30.035131 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:30.035098    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-node-exporter-tls\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:30.037737 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:30.037694    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c7a5a73-a310-4974-9fb9-58220b1ae2c8-node-exporter-tls\") pod \"node-exporter-fv299\" (UID: \"4c7a5a73-a310-4974-9fb9-58220b1ae2c8\") " pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:30.149967 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:30.149856    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:30.190851 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:30.190816    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fv299"
Sep 19 11:35:30.202525 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:35:30.202494    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7a5a73_a310_4974_9fb9_58220b1ae2c8.slice/crio-e59acd7f0a13841fa442908323cbc5ef85f55a0a4e0a6065061166f73a0a438b WatchSource:0}: Error finding container e59acd7f0a13841fa442908323cbc5ef85f55a0a4e0a6065061166f73a0a438b: Status 404 returned error can't find the container with id e59acd7f0a13841fa442908323cbc5ef85f55a0a4e0a6065061166f73a0a438b
Sep 19 11:35:30.249328 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:30.249292    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:30.320359 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:30.320322    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:30.460963 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:30.460863    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:30.561017 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:30.560985    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:30.574628 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:30.574587    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fv299" event={"ID":"4c7a5a73-a310-4974-9fb9-58220b1ae2c8","Type":"ContainerStarted","Data":"e59acd7f0a13841fa442908323cbc5ef85f55a0a4e0a6065061166f73a0a438b"}
Sep 19 11:35:31.580524 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:31.580489    2584 generic.go:356] "Generic (PLEG): container finished" podID="4c7a5a73-a310-4974-9fb9-58220b1ae2c8" containerID="eb1e31e2b69b317facf9ee5d44a20e6dad45f199f09755a6192b849decce37e3" exitCode=0
Sep 19 11:35:31.581016 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:31.580571    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fv299" event={"ID":"4c7a5a73-a310-4974-9fb9-58220b1ae2c8","Type":"ContainerDied","Data":"eb1e31e2b69b317facf9ee5d44a20e6dad45f199f09755a6192b849decce37e3"}
Sep 19 11:35:32.218013 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:32.217974    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:32.318750 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:32.318709    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:32.381557 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:32.381494    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:32.586989 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:32.586947    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fv299" event={"ID":"4c7a5a73-a310-4974-9fb9-58220b1ae2c8","Type":"ContainerStarted","Data":"cc2a5a00347f7ae51c69ec1c69c4497a74223d237aea5607527de8366d256964"}
Sep 19 11:35:32.587406 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:32.587013    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fv299" event={"ID":"4c7a5a73-a310-4974-9fb9-58220b1ae2c8","Type":"ContainerStarted","Data":"1c95837396044da2c50fc48c828583ca731981a1c8f4159ff67e6adb86f858cd"}
Sep 19 11:35:32.607032 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:32.606972    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fv299" podStartSLOduration=2.45856456 podStartE2EDuration="3.606954036s" totalImagesPullingTime="1.073557359s" totalInitContainerRuntime="74.832117ms" isStatefulPod=true podCreationTimestamp="2026-09-19 11:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:35:32.605210179 +0000 UTC m=+116.805239982" watchObservedRunningTime="2026-09-19 11:35:32.606954036 +0000 UTC m=+116.806983833"
Sep 19 11:35:32.679593 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:32.679563    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:32.778647 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:32.778594    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:32.928977 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:32.928844    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:34.059143 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:34.059112    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:34.133732 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:34.133699    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:34.188334 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:34.188291    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:34.289722 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:34.289676    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:34.340429 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:34.340329    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:34.843411 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:34.843374    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:34.894214 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:34.894182    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:35.495035 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:35.495004    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:39.026438 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.026387    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-657b49657d-km2v8" podUID="60111092-bf1d-4949-b3a2-fc36e4a01d59" containerName="registry" containerID="cri-o://a17455a0d65c27cb2b58b268d51c75392641df12a5f91170aa0b17560843ee4b" gracePeriod=30
Sep 19 11:35:39.275304 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.275267    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:35:39.425111 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.425083    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjmxg\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-kube-api-access-bjmxg\") pod \"60111092-bf1d-4949-b3a2-fc36e4a01d59\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") "
Sep 19 11:35:39.425111 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.425117    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60111092-bf1d-4949-b3a2-fc36e4a01d59-installation-pull-secrets\") pod \"60111092-bf1d-4949-b3a2-fc36e4a01d59\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") "
Sep 19 11:35:39.425396 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.425163    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60111092-bf1d-4949-b3a2-fc36e4a01d59-image-registry-private-configuration\") pod \"60111092-bf1d-4949-b3a2-fc36e4a01d59\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") "
Sep 19 11:35:39.425396 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.425220    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60111092-bf1d-4949-b3a2-fc36e4a01d59-trusted-ca\") pod \"60111092-bf1d-4949-b3a2-fc36e4a01d59\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") "
Sep 19 11:35:39.425396 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.425291    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-bound-sa-token\") pod \"60111092-bf1d-4949-b3a2-fc36e4a01d59\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") "
Sep 19 11:35:39.425396 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.425355    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60111092-bf1d-4949-b3a2-fc36e4a01d59-ca-trust-extracted\") pod \"60111092-bf1d-4949-b3a2-fc36e4a01d59\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") "
Sep 19 11:35:39.425396 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.425386    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls\") pod \"60111092-bf1d-4949-b3a2-fc36e4a01d59\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") "
Sep 19 11:35:39.425633 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.425437    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-certificates\") pod \"60111092-bf1d-4949-b3a2-fc36e4a01d59\" (UID: \"60111092-bf1d-4949-b3a2-fc36e4a01d59\") "
Sep 19 11:35:39.425778 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.425711    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60111092-bf1d-4949-b3a2-fc36e4a01d59-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "60111092-bf1d-4949-b3a2-fc36e4a01d59" (UID: "60111092-bf1d-4949-b3a2-fc36e4a01d59"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:35:39.426315 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.426285    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "60111092-bf1d-4949-b3a2-fc36e4a01d59" (UID: "60111092-bf1d-4949-b3a2-fc36e4a01d59"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:35:39.428329 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.428264    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "60111092-bf1d-4949-b3a2-fc36e4a01d59" (UID: "60111092-bf1d-4949-b3a2-fc36e4a01d59"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:35:39.428465 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.428321    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60111092-bf1d-4949-b3a2-fc36e4a01d59-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "60111092-bf1d-4949-b3a2-fc36e4a01d59" (UID: "60111092-bf1d-4949-b3a2-fc36e4a01d59"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:35:39.428465 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.428409    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60111092-bf1d-4949-b3a2-fc36e4a01d59-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "60111092-bf1d-4949-b3a2-fc36e4a01d59" (UID: "60111092-bf1d-4949-b3a2-fc36e4a01d59"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:35:39.428465 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.428409    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "60111092-bf1d-4949-b3a2-fc36e4a01d59" (UID: "60111092-bf1d-4949-b3a2-fc36e4a01d59"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:35:39.428627 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.428569    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-kube-api-access-bjmxg" (OuterVolumeSpecName: "kube-api-access-bjmxg") pod "60111092-bf1d-4949-b3a2-fc36e4a01d59" (UID: "60111092-bf1d-4949-b3a2-fc36e4a01d59"). InnerVolumeSpecName "kube-api-access-bjmxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:35:39.436450 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.436405    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60111092-bf1d-4949-b3a2-fc36e4a01d59-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "60111092-bf1d-4949-b3a2-fc36e4a01d59" (UID: "60111092-bf1d-4949-b3a2-fc36e4a01d59"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:35:39.526852 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.526821    2584 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60111092-bf1d-4949-b3a2-fc36e4a01d59-image-registry-private-configuration\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:35:39.526852 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.526849    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60111092-bf1d-4949-b3a2-fc36e4a01d59-trusted-ca\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:35:39.526852 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.526859    2584 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-bound-sa-token\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:35:39.527150 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.526868    2584 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60111092-bf1d-4949-b3a2-fc36e4a01d59-ca-trust-extracted\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:35:39.527150 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.526877    2584 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-tls\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:35:39.527150 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.526885    2584 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60111092-bf1d-4949-b3a2-fc36e4a01d59-registry-certificates\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:35:39.527150 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.526895    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjmxg\" (UniqueName: \"kubernetes.io/projected/60111092-bf1d-4949-b3a2-fc36e4a01d59-kube-api-access-bjmxg\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:35:39.527150 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.526904    2584 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60111092-bf1d-4949-b3a2-fc36e4a01d59-installation-pull-secrets\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:35:39.614510 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.614477    2584 generic.go:356] "Generic (PLEG): container finished" podID="60111092-bf1d-4949-b3a2-fc36e4a01d59" containerID="a17455a0d65c27cb2b58b268d51c75392641df12a5f91170aa0b17560843ee4b" exitCode=0
Sep 19 11:35:39.614667 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.614524    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657b49657d-km2v8"
Sep 19 11:35:39.614667 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.614538    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657b49657d-km2v8" event={"ID":"60111092-bf1d-4949-b3a2-fc36e4a01d59","Type":"ContainerDied","Data":"a17455a0d65c27cb2b58b268d51c75392641df12a5f91170aa0b17560843ee4b"}
Sep 19 11:35:39.614667 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.614580    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657b49657d-km2v8" event={"ID":"60111092-bf1d-4949-b3a2-fc36e4a01d59","Type":"ContainerDied","Data":"11723254c09d25254fa7f0b08d8761cef95624daca602241443ea1e7b1dd795f"}
Sep 19 11:35:39.614667 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.614602    2584 scope.go:117] "RemoveContainer" containerID="a17455a0d65c27cb2b58b268d51c75392641df12a5f91170aa0b17560843ee4b"
Sep 19 11:35:39.623533 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.623514    2584 scope.go:117] "RemoveContainer" containerID="a17455a0d65c27cb2b58b268d51c75392641df12a5f91170aa0b17560843ee4b"
Sep 19 11:35:39.623865 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:35:39.623844    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17455a0d65c27cb2b58b268d51c75392641df12a5f91170aa0b17560843ee4b\": container with ID starting with a17455a0d65c27cb2b58b268d51c75392641df12a5f91170aa0b17560843ee4b not found: ID does not exist" containerID="a17455a0d65c27cb2b58b268d51c75392641df12a5f91170aa0b17560843ee4b"
Sep 19 11:35:39.624008 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.623880    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17455a0d65c27cb2b58b268d51c75392641df12a5f91170aa0b17560843ee4b"} err="failed to get container status \"a17455a0d65c27cb2b58b268d51c75392641df12a5f91170aa0b17560843ee4b\": rpc error: code = NotFound desc = could not find container \"a17455a0d65c27cb2b58b268d51c75392641df12a5f91170aa0b17560843ee4b\": container with ID starting with a17455a0d65c27cb2b58b268d51c75392641df12a5f91170aa0b17560843ee4b not found: ID does not exist"
Sep 19 11:35:39.636313 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.636278    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-657b49657d-km2v8"]
Sep 19 11:35:39.639545 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.639517    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-657b49657d-km2v8"]
Sep 19 11:35:39.904137 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:39.904105    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60111092-bf1d-4949-b3a2-fc36e4a01d59" path="/var/lib/kubelet/pods/60111092-bf1d-4949-b3a2-fc36e4a01d59/volumes"
Sep 19 11:35:43.460041 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:43.459998    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:43.509959 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:43.509928    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:47.110009 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:47.109978    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:47.159614 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:47.159582    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:47.651409 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:47.651366    2584 generic.go:356] "Generic (PLEG): container finished" podID="cc7e845c-b6fd-4a96-8258-80b4e8f02e57" containerID="2ad883109fbf14373ac9fe9328dc64fc92ef0e4de4294373d9b37238d803e722" exitCode=0
Sep 19 11:35:47.651638 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:47.651444    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd" event={"ID":"cc7e845c-b6fd-4a96-8258-80b4e8f02e57","Type":"ContainerDied","Data":"2ad883109fbf14373ac9fe9328dc64fc92ef0e4de4294373d9b37238d803e722"}
Sep 19 11:35:47.651927 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:47.651899    2584 scope.go:117] "RemoveContainer" containerID="2ad883109fbf14373ac9fe9328dc64fc92ef0e4de4294373d9b37238d803e722"
Sep 19 11:35:47.697170 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:47.697127    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs\") pod \"network-metrics-daemon-pgph6\" (UID: \"83707003-2551-42b5-8943-8243dd6daa80\") " pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:35:47.699673 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:47.699528    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83707003-2551-42b5-8943-8243dd6daa80-metrics-certs\") pod \"network-metrics-daemon-pgph6\" (UID: \"83707003-2551-42b5-8943-8243dd6daa80\") " pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:35:47.918897 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:47.918741    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6c6f7\""
Sep 19 11:35:47.925964 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:47.925936    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgph6"
Sep 19 11:35:48.203642 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:48.203560    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pgph6"]
Sep 19 11:35:48.206559 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:35:48.206527    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83707003_2551_42b5_8943_8243dd6daa80.slice/crio-620e12cdcc44207177858db009142c0cb265617906251e23fc17f312c920ea67 WatchSource:0}: Error finding container 620e12cdcc44207177858db009142c0cb265617906251e23fc17f312c920ea67: Status 404 returned error can't find the container with id 620e12cdcc44207177858db009142c0cb265617906251e23fc17f312c920ea67
Sep 19 11:35:48.586330 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:48.586303    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-687866bb-9xvr5_5b388e80-43fb-4367-a7c1-6c53f04caffa/router/0.log"
Sep 19 11:35:48.597885 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:48.597855    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h4xjv_2066d78e-223c-4531-98aa-edc826b41c3b/serve-healthcheck-canary/0.log"
Sep 19 11:35:48.658125 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:48.658086    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-7wgwd" event={"ID":"cc7e845c-b6fd-4a96-8258-80b4e8f02e57","Type":"ContainerStarted","Data":"2fcfdae1235467bba09c5d33c93f8446caa5317bdfa80ef2731051242bdaa7bc"}
Sep 19 11:35:48.659444 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:48.659413    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pgph6" event={"ID":"83707003-2551-42b5-8943-8243dd6daa80","Type":"ContainerStarted","Data":"620e12cdcc44207177858db009142c0cb265617906251e23fc17f312c920ea67"}
Sep 19 11:35:49.665123 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:49.665087    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pgph6" event={"ID":"83707003-2551-42b5-8943-8243dd6daa80","Type":"ContainerStarted","Data":"e82f5e35ac2b7f399ab42eba069686491bcdd7a9e6d90894baea984d24a7c3c5"}
Sep 19 11:35:49.665651 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:49.665143    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pgph6" event={"ID":"83707003-2551-42b5-8943-8243dd6daa80","Type":"ContainerStarted","Data":"589deba8c58d2e51fe18be5c2f38c28e7272646c1a542bcc8c9db382555081e0"}
Sep 19 11:35:49.680380 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:49.680327    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pgph6" podStartSLOduration=130.714740714 podStartE2EDuration="2m11.680308191s" totalImagesPullingTime="965.567477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:35:49.679018711 +0000 UTC m=+133.879048513" watchObservedRunningTime="2026-09-19 11:35:49.680308191 +0000 UTC m=+133.880338044"
Sep 19 11:35:49.757290 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:49.757247    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:49.812102 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:49.812059    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:50.278285 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:50.278246    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:50.328244 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:50.328205    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:50.382216 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:50.382183    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:50.514445 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:50.514391    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:50.614673 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:50.614623    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:50.664687 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:50.664646    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:51.362840 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:51.362797    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg" podUID="c323067d-ee96-4a29-8322-83ce5f3f454f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 19 11:35:51.674333 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:51.674301    2584 generic.go:356] "Generic (PLEG): container finished" podID="aa857849-d4f4-4815-aa2d-8412e4a8033a" containerID="1641d488eaf60ff45ba80ad3082a4877965b3cb320ca4d5e469cad620540d1a7" exitCode=0
Sep 19 11:35:51.674523 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:51.674390    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ftf42" event={"ID":"aa857849-d4f4-4815-aa2d-8412e4a8033a","Type":"ContainerDied","Data":"1641d488eaf60ff45ba80ad3082a4877965b3cb320ca4d5e469cad620540d1a7"}
Sep 19 11:35:51.674781 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:51.674753    2584 scope.go:117] "RemoveContainer" containerID="1641d488eaf60ff45ba80ad3082a4877965b3cb320ca4d5e469cad620540d1a7"
Sep 19 11:35:51.676094 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:51.676071    2584 generic.go:356] "Generic (PLEG): container finished" podID="96c0a054-f70d-46dd-b510-9056a47a6b9b" containerID="f1035eab25dc4bcc617f0870e9b6756e852dcde0e1c6c1bd5075b76bedb5df5a" exitCode=0
Sep 19 11:35:51.676150 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:51.676115    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-m9jqg" event={"ID":"96c0a054-f70d-46dd-b510-9056a47a6b9b","Type":"ContainerDied","Data":"f1035eab25dc4bcc617f0870e9b6756e852dcde0e1c6c1bd5075b76bedb5df5a"}
Sep 19 11:35:51.676423 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:51.676406    2584 scope.go:117] "RemoveContainer" containerID="f1035eab25dc4bcc617f0870e9b6756e852dcde0e1c6c1bd5075b76bedb5df5a"
Sep 19 11:35:52.681986 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:52.681943    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ftf42" event={"ID":"aa857849-d4f4-4815-aa2d-8412e4a8033a","Type":"ContainerStarted","Data":"5c9bd4565b0a5eb7d3923abefdddd6fc8befbbfce5f11dc33ea57f330b758710"}
Sep 19 11:35:52.683627 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:52.683606    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-m9jqg" event={"ID":"96c0a054-f70d-46dd-b510-9056a47a6b9b","Type":"ContainerStarted","Data":"6ccbd73ef2b3adccf7b994bacdf147bf50c4737bd0e5bf98e7f1bb3fac77eccd"}
Sep 19 11:35:57.433472 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:57.433436    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:58.458414 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:58.458371    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:35:58.523412 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:35:58.523372    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:36:01.364737 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:36:01.364682    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg" podUID="c323067d-ee96-4a29-8322-83ce5f3f454f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 19 11:36:11.362534 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:36:11.362491    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg" podUID="c323067d-ee96-4a29-8322-83ce5f3f454f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 19 11:36:11.363087 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:36:11.362576    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg"
Sep 19 11:36:11.363151 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:36:11.363139    2584 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"84dc6ed33d66de0ef1157a4a69069f02ac98e7b870455273ff1cf5485b5f4a2f"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 19 11:36:11.363188 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:36:11.363179    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg" podUID="c323067d-ee96-4a29-8322-83ce5f3f454f" containerName="service-proxy" containerID="cri-o://84dc6ed33d66de0ef1157a4a69069f02ac98e7b870455273ff1cf5485b5f4a2f" gracePeriod=30
Sep 19 11:36:11.766547 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:36:11.766450    2584 generic.go:356] "Generic (PLEG): container finished" podID="c323067d-ee96-4a29-8322-83ce5f3f454f" containerID="84dc6ed33d66de0ef1157a4a69069f02ac98e7b870455273ff1cf5485b5f4a2f" exitCode=2
Sep 19 11:36:11.766547 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:36:11.766486    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg" event={"ID":"c323067d-ee96-4a29-8322-83ce5f3f454f","Type":"ContainerDied","Data":"84dc6ed33d66de0ef1157a4a69069f02ac98e7b870455273ff1cf5485b5f4a2f"}
Sep 19 11:36:11.766547 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:36:11.766546    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cc5f998dc-9cvsg" event={"ID":"c323067d-ee96-4a29-8322-83ce5f3f454f","Type":"ContainerStarted","Data":"c056c225d87ec66da8e9b2b2d72a0eb5fa70fd5d22a0f3ff62fc1ba9d3ed0652"}
Sep 19 11:36:13.460964 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:36:13.460932    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:36:13.514409 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:36:13.514379    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:36:32.320216 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:36:32.320176    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:36:45.345678 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:36:45.345632    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:36:45.399454 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:36:45.399420    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:36:53.756247 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:36:53.756214    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:36:53.823632 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:36:53.823596    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:36:57.247414 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:36:57.247382    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:36:57.300434 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:36:57.300393    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:04.607886 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:04.607854    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:04.708640 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:04.708596    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:04.759712 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:04.759675    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:13.460658 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:13.460618    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:13.560793 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:13.560759    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:18.606150 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:18.606114    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:47.590266 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:47.590224    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:47.703350 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:47.703317    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:47.802862 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:47.802782    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:48.821865 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:48.821833    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:48.877784 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:48.877751    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:49.175843 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:49.175753    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:49.447728 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:49.447641    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:49.755754 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:49.755678    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:53.710804 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:53.710773    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:53.809123 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:53.809070    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:53.907985 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:53.907948    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:53.980167 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:53.975674    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:54.076724 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:54.076692    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:54.437028 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:54.436985    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:37:55.511720 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:55.511689    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv"]
Sep 19 11:37:55.512158 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:55.512020    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60111092-bf1d-4949-b3a2-fc36e4a01d59" containerName="registry"
Sep 19 11:37:55.512158 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:55.512032    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="60111092-bf1d-4949-b3a2-fc36e4a01d59" containerName="registry"
Sep 19 11:37:55.512158 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:55.512078    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="60111092-bf1d-4949-b3a2-fc36e4a01d59" containerName="registry"
Sep 19 11:37:55.515138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:55.515121    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv"
Sep 19 11:37:55.517719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:55.517693    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 19 11:37:55.517719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:55.517706    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f9x6f\""
Sep 19 11:37:55.517719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:55.517704    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 19 11:37:55.522026 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:55.522002    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv"]
Sep 19 11:37:55.620148 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:55.620099    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f4b7f90-6390-4646-8b4d-158951980d9a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv\" (UID: \"5f4b7f90-6390-4646-8b4d-158951980d9a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv"
Sep 19 11:37:55.620148 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:55.620156    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f4b7f90-6390-4646-8b4d-158951980d9a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv\" (UID: \"5f4b7f90-6390-4646-8b4d-158951980d9a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv"
Sep 19 11:37:55.620406 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:55.620239    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qxrk\" (UniqueName: \"kubernetes.io/projected/5f4b7f90-6390-4646-8b4d-158951980d9a-kube-api-access-2qxrk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv\" (UID: \"5f4b7f90-6390-4646-8b4d-158951980d9a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv"
Sep 19 11:37:55.721458 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:55.721415    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qxrk\" (UniqueName: \"kubernetes.io/projected/5f4b7f90-6390-4646-8b4d-158951980d9a-kube-api-access-2qxrk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv\" (UID: \"5f4b7f90-6390-4646-8b4d-158951980d9a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv"
Sep 19 11:37:55.721690 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:55.721520    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f4b7f90-6390-4646-8b4d-158951980d9a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv\" (UID: \"5f4b7f90-6390-4646-8b4d-158951980d9a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv"
Sep 19 11:37:55.721690 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:55.721558    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f4b7f90-6390-4646-8b4d-158951980d9a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv\" (UID: \"5f4b7f90-6390-4646-8b4d-158951980d9a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv"
Sep 19 11:37:55.722052 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:55.722013    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f4b7f90-6390-4646-8b4d-158951980d9a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv\" (UID: \"5f4b7f90-6390-4646-8b4d-158951980d9a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv"
Sep 19 11:37:55.722052 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:55.722048    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f4b7f90-6390-4646-8b4d-158951980d9a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv\" (UID: \"5f4b7f90-6390-4646-8b4d-158951980d9a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv"
Sep 19 11:37:55.730884 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:55.730856    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qxrk\" (UniqueName: \"kubernetes.io/projected/5f4b7f90-6390-4646-8b4d-158951980d9a-kube-api-access-2qxrk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv\" (UID: \"5f4b7f90-6390-4646-8b4d-158951980d9a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv"
Sep 19 11:37:55.826262 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:55.826226    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv"
Sep 19 11:37:56.097651 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:56.097620    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv"]
Sep 19 11:37:56.100305 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:37:56.100270    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4b7f90_6390_4646_8b4d_158951980d9a.slice/crio-1bfe44ae0c93c9dc2fa3a343499eba7ba1fe81e339bb7f056bf2dfdb6e8bdc0d WatchSource:0}: Error finding container 1bfe44ae0c93c9dc2fa3a343499eba7ba1fe81e339bb7f056bf2dfdb6e8bdc0d: Status 404 returned error can't find the container with id 1bfe44ae0c93c9dc2fa3a343499eba7ba1fe81e339bb7f056bf2dfdb6e8bdc0d
Sep 19 11:37:56.166209 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:56.166172    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv" event={"ID":"5f4b7f90-6390-4646-8b4d-158951980d9a","Type":"ContainerStarted","Data":"1bfe44ae0c93c9dc2fa3a343499eba7ba1fe81e339bb7f056bf2dfdb6e8bdc0d"}
Sep 19 11:37:56.732570 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:37:56.732536    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:02.193812 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:02.193770    2584 generic.go:356] "Generic (PLEG): container finished" podID="5f4b7f90-6390-4646-8b4d-158951980d9a" containerID="8ea99e9350d428e981ce0c95b6e5b91c52afd7cc552d566480f0a052276e895c" exitCode=0
Sep 19 11:38:02.194232 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:02.193859    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv" event={"ID":"5f4b7f90-6390-4646-8b4d-158951980d9a","Type":"ContainerDied","Data":"8ea99e9350d428e981ce0c95b6e5b91c52afd7cc552d566480f0a052276e895c"}
Sep 19 11:38:03.200089 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:03.200048    2584 generic.go:356] "Generic (PLEG): container finished" podID="5f4b7f90-6390-4646-8b4d-158951980d9a" containerID="a207377fea285e170ae3f55978ac4ccff227580d5ea7ae9e99250299baa20e29" exitCode=0
Sep 19 11:38:03.200469 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:03.200139    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv" event={"ID":"5f4b7f90-6390-4646-8b4d-158951980d9a","Type":"ContainerDied","Data":"a207377fea285e170ae3f55978ac4ccff227580d5ea7ae9e99250299baa20e29"}
Sep 19 11:38:09.230052 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:09.230022    2584 generic.go:356] "Generic (PLEG): container finished" podID="5f4b7f90-6390-4646-8b4d-158951980d9a" containerID="53e45b4798ede6a00d43a17f732db3a21e6063931e21481f6cbfb912b635eaf6" exitCode=0
Sep 19 11:38:09.230455 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:09.230105    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv" event={"ID":"5f4b7f90-6390-4646-8b4d-158951980d9a","Type":"ContainerDied","Data":"53e45b4798ede6a00d43a17f732db3a21e6063931e21481f6cbfb912b635eaf6"}
Sep 19 11:38:10.360790 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:10.360764    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv"
Sep 19 11:38:10.448960 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:10.448886    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qxrk\" (UniqueName: \"kubernetes.io/projected/5f4b7f90-6390-4646-8b4d-158951980d9a-kube-api-access-2qxrk\") pod \"5f4b7f90-6390-4646-8b4d-158951980d9a\" (UID: \"5f4b7f90-6390-4646-8b4d-158951980d9a\") "
Sep 19 11:38:10.449152 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:10.448974    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f4b7f90-6390-4646-8b4d-158951980d9a-util\") pod \"5f4b7f90-6390-4646-8b4d-158951980d9a\" (UID: \"5f4b7f90-6390-4646-8b4d-158951980d9a\") "
Sep 19 11:38:10.449152 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:10.449077    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f4b7f90-6390-4646-8b4d-158951980d9a-bundle\") pod \"5f4b7f90-6390-4646-8b4d-158951980d9a\" (UID: \"5f4b7f90-6390-4646-8b4d-158951980d9a\") "
Sep 19 11:38:10.449515 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:10.449491    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4b7f90-6390-4646-8b4d-158951980d9a-bundle" (OuterVolumeSpecName: "bundle") pod "5f4b7f90-6390-4646-8b4d-158951980d9a" (UID: "5f4b7f90-6390-4646-8b4d-158951980d9a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:38:10.451532 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:10.451492    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4b7f90-6390-4646-8b4d-158951980d9a-kube-api-access-2qxrk" (OuterVolumeSpecName: "kube-api-access-2qxrk") pod "5f4b7f90-6390-4646-8b4d-158951980d9a" (UID: "5f4b7f90-6390-4646-8b4d-158951980d9a"). InnerVolumeSpecName "kube-api-access-2qxrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:38:10.453149 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:10.453125    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4b7f90-6390-4646-8b4d-158951980d9a-util" (OuterVolumeSpecName: "util") pod "5f4b7f90-6390-4646-8b4d-158951980d9a" (UID: "5f4b7f90-6390-4646-8b4d-158951980d9a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:38:10.550288 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:10.550198    2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f4b7f90-6390-4646-8b4d-158951980d9a-bundle\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:38:10.550288 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:10.550229    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qxrk\" (UniqueName: \"kubernetes.io/projected/5f4b7f90-6390-4646-8b4d-158951980d9a-kube-api-access-2qxrk\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:38:10.550288 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:10.550240    2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f4b7f90-6390-4646-8b4d-158951980d9a-util\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:38:11.082820 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.082790    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:11.197041 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.196996    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:11.240879 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.240854    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv"
Sep 19 11:38:11.241062 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.240854    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9fcv" event={"ID":"5f4b7f90-6390-4646-8b4d-158951980d9a","Type":"ContainerDied","Data":"1bfe44ae0c93c9dc2fa3a343499eba7ba1fe81e339bb7f056bf2dfdb6e8bdc0d"}
Sep 19 11:38:11.241062 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.240964    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bfe44ae0c93c9dc2fa3a343499eba7ba1fe81e339bb7f056bf2dfdb6e8bdc0d"
Sep 19 11:38:11.736752 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.736719    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:11.863779 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.863739    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"]
Sep 19 11:38:11.864066 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.864053    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f4b7f90-6390-4646-8b4d-158951980d9a" containerName="pull"
Sep 19 11:38:11.864122 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.864068    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f4b7f90-6390-4646-8b4d-158951980d9a" containerName="pull"
Sep 19 11:38:11.864122 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.864090    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f4b7f90-6390-4646-8b4d-158951980d9a" containerName="util"
Sep 19 11:38:11.864122 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.864096    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f4b7f90-6390-4646-8b4d-158951980d9a" containerName="util"
Sep 19 11:38:11.864122 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.864101    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f4b7f90-6390-4646-8b4d-158951980d9a" containerName="extract"
Sep 19 11:38:11.864122 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.864107    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f4b7f90-6390-4646-8b4d-158951980d9a" containerName="extract"
Sep 19 11:38:11.864272 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.864156    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f4b7f90-6390-4646-8b4d-158951980d9a" containerName="extract"
Sep 19 11:38:11.866187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.866172    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:11.868755 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.868722    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 19 11:38:11.868755 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.868727    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 19 11:38:11.868974 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.868945    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-cjw94\""
Sep 19 11:38:11.869083 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.869066    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Sep 19 11:38:11.869203 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.869135    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Sep 19 11:38:11.869985 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.869968    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Sep 19 11:38:11.883631 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.883604    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"]
Sep 19 11:38:11.961080 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.961048    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/549eeb37-540c-4280-9b6d-87a08afaddfa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9f59f68b4-vx8jw\" (UID: \"549eeb37-540c-4280-9b6d-87a08afaddfa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:11.961080 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.961083    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/549eeb37-540c-4280-9b6d-87a08afaddfa-tmp\") pod \"opendatahub-operator-controller-manager-9f59f68b4-vx8jw\" (UID: \"549eeb37-540c-4280-9b6d-87a08afaddfa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:11.961294 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.961110    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/549eeb37-540c-4280-9b6d-87a08afaddfa-webhook-cert\") pod \"opendatahub-operator-controller-manager-9f59f68b4-vx8jw\" (UID: \"549eeb37-540c-4280-9b6d-87a08afaddfa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:11.961294 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.961127    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/549eeb37-540c-4280-9b6d-87a08afaddfa-metrics-cert\") pod \"opendatahub-operator-controller-manager-9f59f68b4-vx8jw\" (UID: \"549eeb37-540c-4280-9b6d-87a08afaddfa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:11.961294 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.961204    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/549eeb37-540c-4280-9b6d-87a08afaddfa-manifests\") pod \"opendatahub-operator-controller-manager-9f59f68b4-vx8jw\" (UID: \"549eeb37-540c-4280-9b6d-87a08afaddfa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:11.961294 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:11.961242    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgwn7\" (UniqueName: \"kubernetes.io/projected/549eeb37-540c-4280-9b6d-87a08afaddfa-kube-api-access-fgwn7\") pod \"opendatahub-operator-controller-manager-9f59f68b4-vx8jw\" (UID: \"549eeb37-540c-4280-9b6d-87a08afaddfa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:12.016470 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:12.016379    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:12.062514 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:12.062477    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/549eeb37-540c-4280-9b6d-87a08afaddfa-manifests\") pod \"opendatahub-operator-controller-manager-9f59f68b4-vx8jw\" (UID: \"549eeb37-540c-4280-9b6d-87a08afaddfa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:12.062514 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:12.062516    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgwn7\" (UniqueName: \"kubernetes.io/projected/549eeb37-540c-4280-9b6d-87a08afaddfa-kube-api-access-fgwn7\") pod \"opendatahub-operator-controller-manager-9f59f68b4-vx8jw\" (UID: \"549eeb37-540c-4280-9b6d-87a08afaddfa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:12.062711 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:12.062546    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/549eeb37-540c-4280-9b6d-87a08afaddfa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9f59f68b4-vx8jw\" (UID: \"549eeb37-540c-4280-9b6d-87a08afaddfa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:12.062711 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:12.062653    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/549eeb37-540c-4280-9b6d-87a08afaddfa-tmp\") pod \"opendatahub-operator-controller-manager-9f59f68b4-vx8jw\" (UID: \"549eeb37-540c-4280-9b6d-87a08afaddfa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:12.062788 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:12.062710    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/549eeb37-540c-4280-9b6d-87a08afaddfa-webhook-cert\") pod \"opendatahub-operator-controller-manager-9f59f68b4-vx8jw\" (UID: \"549eeb37-540c-4280-9b6d-87a08afaddfa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:12.062788 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:12.062740    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/549eeb37-540c-4280-9b6d-87a08afaddfa-metrics-cert\") pod \"opendatahub-operator-controller-manager-9f59f68b4-vx8jw\" (UID: \"549eeb37-540c-4280-9b6d-87a08afaddfa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:12.063002 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:12.062985    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/549eeb37-540c-4280-9b6d-87a08afaddfa-manifests\") pod \"opendatahub-operator-controller-manager-9f59f68b4-vx8jw\" (UID: \"549eeb37-540c-4280-9b6d-87a08afaddfa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:12.065369 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:12.065334    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/549eeb37-540c-4280-9b6d-87a08afaddfa-tmp\") pod \"opendatahub-operator-controller-manager-9f59f68b4-vx8jw\" (UID: \"549eeb37-540c-4280-9b6d-87a08afaddfa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:12.065369 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:12.065353    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/549eeb37-540c-4280-9b6d-87a08afaddfa-webhook-cert\") pod \"opendatahub-operator-controller-manager-9f59f68b4-vx8jw\" (UID: \"549eeb37-540c-4280-9b6d-87a08afaddfa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:12.065562 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:12.065437    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/549eeb37-540c-4280-9b6d-87a08afaddfa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9f59f68b4-vx8jw\" (UID: \"549eeb37-540c-4280-9b6d-87a08afaddfa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:12.065618 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:12.065586    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/549eeb37-540c-4280-9b6d-87a08afaddfa-metrics-cert\") pod \"opendatahub-operator-controller-manager-9f59f68b4-vx8jw\" (UID: \"549eeb37-540c-4280-9b6d-87a08afaddfa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:12.079364 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:12.079335    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgwn7\" (UniqueName: \"kubernetes.io/projected/549eeb37-540c-4280-9b6d-87a08afaddfa-kube-api-access-fgwn7\") pod \"opendatahub-operator-controller-manager-9f59f68b4-vx8jw\" (UID: \"549eeb37-540c-4280-9b6d-87a08afaddfa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:12.119852 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:12.119817    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:12.177039 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:12.176992    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:12.222197 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:12.222159    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:12.320875 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:12.320845    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:12.455568 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:12.455533    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"]
Sep 19 11:38:12.459217 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:38:12.459185    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549eeb37_540c_4280_9b6d_87a08afaddfa.slice/crio-0948de03bfaf3d4617514121c1aafc775b12b03deccb0d4cd604520e5cb06ddb WatchSource:0}: Error finding container 0948de03bfaf3d4617514121c1aafc775b12b03deccb0d4cd604520e5cb06ddb: Status 404 returned error can't find the container with id 0948de03bfaf3d4617514121c1aafc775b12b03deccb0d4cd604520e5cb06ddb
Sep 19 11:38:13.252257 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:13.252218    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw" event={"ID":"549eeb37-540c-4280-9b6d-87a08afaddfa","Type":"ContainerStarted","Data":"0948de03bfaf3d4617514121c1aafc775b12b03deccb0d4cd604520e5cb06ddb"}
Sep 19 11:38:14.683378 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:14.683335    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:14.789657 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:14.789621    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:16.270698 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:16.270662    2584 generic.go:356] "Generic (PLEG): container finished" podID="549eeb37-540c-4280-9b6d-87a08afaddfa" containerID="1f8b077ef29221fea0eccbf09ec951e274587390925b73b5c44ebafb8b79f126" exitCode=0
Sep 19 11:38:16.271246 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:16.270742    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw" event={"ID":"549eeb37-540c-4280-9b6d-87a08afaddfa","Type":"ContainerDied","Data":"1f8b077ef29221fea0eccbf09ec951e274587390925b73b5c44ebafb8b79f126"}
Sep 19 11:38:16.895922 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:16.895890    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:16.969924 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:16.969889    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:17.277191 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:17.277104    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw" event={"ID":"549eeb37-540c-4280-9b6d-87a08afaddfa","Type":"ContainerStarted","Data":"d8791b1975b3079530feed99595490604a9e6c33a83228f12eb91a1e2b790472"}
Sep 19 11:38:17.277642 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:17.277247    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:17.301700 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:17.301637    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw" podStartSLOduration=2.760568077 podStartE2EDuration="6.30162098s" totalImagesPullingTime="3.053665392s" totalInitContainerRuntime="487.387511ms" isStatefulPod=false podCreationTimestamp="2026-09-19 11:38:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:38:17.299901324 +0000 UTC m=+281.499931126" watchObservedRunningTime="2026-09-19 11:38:17.30162098 +0000 UTC m=+281.501650783"
Sep 19 11:38:20.660491 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:20.660456    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:24.004872 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:24.004833    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-82kgm"]
Sep 19 11:38:24.010263 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:24.010245    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-82kgm"
Sep 19 11:38:24.013164 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:24.013141    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 19 11:38:24.013300 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:24.013167    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 19 11:38:24.013300 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:24.013202    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-p4dxb\""
Sep 19 11:38:24.020427 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:24.020400    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-82kgm"]
Sep 19 11:38:24.064456 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:24.064417    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q22w\" (UniqueName: \"kubernetes.io/projected/5388f3ec-bb2a-4707-b52b-70b197a4f3b3-kube-api-access-6q22w\") pod \"cert-manager-operator-controller-manager-684f79c9b9-82kgm\" (UID: \"5388f3ec-bb2a-4707-b52b-70b197a4f3b3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-82kgm"
Sep 19 11:38:24.064657 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:24.064526    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5388f3ec-bb2a-4707-b52b-70b197a4f3b3-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-82kgm\" (UID: \"5388f3ec-bb2a-4707-b52b-70b197a4f3b3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-82kgm"
Sep 19 11:38:24.165252 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:24.165214    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5388f3ec-bb2a-4707-b52b-70b197a4f3b3-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-82kgm\" (UID: \"5388f3ec-bb2a-4707-b52b-70b197a4f3b3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-82kgm"
Sep 19 11:38:24.165434 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:24.165278    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q22w\" (UniqueName: \"kubernetes.io/projected/5388f3ec-bb2a-4707-b52b-70b197a4f3b3-kube-api-access-6q22w\") pod \"cert-manager-operator-controller-manager-684f79c9b9-82kgm\" (UID: \"5388f3ec-bb2a-4707-b52b-70b197a4f3b3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-82kgm"
Sep 19 11:38:24.165634 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:24.165613    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5388f3ec-bb2a-4707-b52b-70b197a4f3b3-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-82kgm\" (UID: \"5388f3ec-bb2a-4707-b52b-70b197a4f3b3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-82kgm"
Sep 19 11:38:24.174272 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:24.174246    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q22w\" (UniqueName: \"kubernetes.io/projected/5388f3ec-bb2a-4707-b52b-70b197a4f3b3-kube-api-access-6q22w\") pod \"cert-manager-operator-controller-manager-684f79c9b9-82kgm\" (UID: \"5388f3ec-bb2a-4707-b52b-70b197a4f3b3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-82kgm"
Sep 19 11:38:24.319335 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:24.319306    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-82kgm"
Sep 19 11:38:24.596813 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:24.596783    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-82kgm"]
Sep 19 11:38:24.597901 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:38:24.597876    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5388f3ec_bb2a_4707_b52b_70b197a4f3b3.slice/crio-8d6f6c92f43a5a6137ab3004f19019382734f1db4aff743187a8f47afe1d51bc WatchSource:0}: Error finding container 8d6f6c92f43a5a6137ab3004f19019382734f1db4aff743187a8f47afe1d51bc: Status 404 returned error can't find the container with id 8d6f6c92f43a5a6137ab3004f19019382734f1db4aff743187a8f47afe1d51bc
Sep 19 11:38:25.312925 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:25.312866    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-82kgm" event={"ID":"5388f3ec-bb2a-4707-b52b-70b197a4f3b3","Type":"ContainerStarted","Data":"8d6f6c92f43a5a6137ab3004f19019382734f1db4aff743187a8f47afe1d51bc"}
Sep 19 11:38:26.168304 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:26.168270    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:27.267099 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:27.267069    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:28.284863 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:28.284819    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9f59f68b4-vx8jw"
Sep 19 11:38:29.264036 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:29.263994    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:30.266866 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:30.266788    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:30.340728 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:30.340693    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-82kgm" event={"ID":"5388f3ec-bb2a-4707-b52b-70b197a4f3b3","Type":"ContainerStarted","Data":"4a9db9d35ccc1d92eb1a778621277b767b1ec48ea37d37e1c9675f51b988b577"}
Sep 19 11:38:30.362057 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:30.362006    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-82kgm" podStartSLOduration=1.965011021 podStartE2EDuration="7.361989905s" totalImagesPullingTime="5.396978884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:38:30.360150653 +0000 UTC m=+294.560180470" watchObservedRunningTime="2026-09-19 11:38:30.361989905 +0000 UTC m=+294.562019707"
Sep 19 11:38:30.884362 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:30.884331    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:30.991399 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:30.991358    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:31.538177 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:31.538144    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:33.252167 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:33.252135    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:33.462754 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:33.462723    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:36.266781 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:36.266748    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:36.325292 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:36.325258    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:37.575636 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:37.575596    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:37.687385 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:37.687338    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:37.827182 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:37.827111    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/1.log"
Sep 19 11:38:37.831794 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:37.831771    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/1.log"
Sep 19 11:38:37.832782 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:37.832764    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6th98_2fed27e0-833f-49a8-b11e-8f61df6e6bd5/ovn-acl-logging/0.log"
Sep 19 11:38:37.837451 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:37.837429    2584 kubelet.go:1648] "Image garbage collection succeeded"
Sep 19 11:38:37.837885 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:37.837872    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6th98_2fed27e0-833f-49a8-b11e-8f61df6e6bd5/ovn-acl-logging/0.log"
Sep 19 11:38:37.990173 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:37.990014    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:38.027936 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.027881    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-k668c"]
Sep 19 11:38:38.030891 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.030873    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-k668c"
Sep 19 11:38:38.033560 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.033539    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 19 11:38:38.033692 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.033540    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-bxttj\""
Sep 19 11:38:38.034823 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.034805    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 19 11:38:38.040822 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.040800    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-k668c"]
Sep 19 11:38:38.079480 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.079395    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9fad1905-a4c8-4520-b0b9-2d1db1fd6624-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-k668c\" (UID: \"9fad1905-a4c8-4520-b0b9-2d1db1fd6624\") " pod="cert-manager/cert-manager-webhook-6486ffd997-k668c"
Sep 19 11:38:38.079480 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.079450    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcv9n\" (UniqueName: \"kubernetes.io/projected/9fad1905-a4c8-4520-b0b9-2d1db1fd6624-kube-api-access-hcv9n\") pod \"cert-manager-webhook-6486ffd997-k668c\" (UID: \"9fad1905-a4c8-4520-b0b9-2d1db1fd6624\") " pod="cert-manager/cert-manager-webhook-6486ffd997-k668c"
Sep 19 11:38:38.087734 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.087708    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:38.180028 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.179984    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9fad1905-a4c8-4520-b0b9-2d1db1fd6624-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-k668c\" (UID: \"9fad1905-a4c8-4520-b0b9-2d1db1fd6624\") " pod="cert-manager/cert-manager-webhook-6486ffd997-k668c"
Sep 19 11:38:38.180200 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.180048    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcv9n\" (UniqueName: \"kubernetes.io/projected/9fad1905-a4c8-4520-b0b9-2d1db1fd6624-kube-api-access-hcv9n\") pod \"cert-manager-webhook-6486ffd997-k668c\" (UID: \"9fad1905-a4c8-4520-b0b9-2d1db1fd6624\") " pod="cert-manager/cert-manager-webhook-6486ffd997-k668c"
Sep 19 11:38:38.200109 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.200074    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:38.203650 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.203618    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9fad1905-a4c8-4520-b0b9-2d1db1fd6624-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-k668c\" (UID: \"9fad1905-a4c8-4520-b0b9-2d1db1fd6624\") " pod="cert-manager/cert-manager-webhook-6486ffd997-k668c"
Sep 19 11:38:38.203790 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.203770    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcv9n\" (UniqueName: \"kubernetes.io/projected/9fad1905-a4c8-4520-b0b9-2d1db1fd6624-kube-api-access-hcv9n\") pod \"cert-manager-webhook-6486ffd997-k668c\" (UID: \"9fad1905-a4c8-4520-b0b9-2d1db1fd6624\") " pod="cert-manager/cert-manager-webhook-6486ffd997-k668c"
Sep 19 11:38:38.338745 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.338661    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-864788f75f-x6j8q"]
Sep 19 11:38:38.340253 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.340233    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-k668c"
Sep 19 11:38:38.341908 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.341892    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-864788f75f-x6j8q"
Sep 19 11:38:38.345444 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.345414    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 19 11:38:38.345599 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.345582    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wqfnh\""
Sep 19 11:38:38.345839 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.345824    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 19 11:38:38.362712 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.362688    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:38.363145 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.363116    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-864788f75f-x6j8q"]
Sep 19 11:38:38.471156 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.471123    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:38.483247 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.483221    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb4pt\" (UniqueName: \"kubernetes.io/projected/f5e9cbc1-01a5-4d28-8287-f100f26a76af-kube-api-access-lb4pt\") pod \"kube-auth-proxy-864788f75f-x6j8q\" (UID: \"f5e9cbc1-01a5-4d28-8287-f100f26a76af\") " pod="openshift-ingress/kube-auth-proxy-864788f75f-x6j8q"
Sep 19 11:38:38.483404 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.483276    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5e9cbc1-01a5-4d28-8287-f100f26a76af-tmp\") pod \"kube-auth-proxy-864788f75f-x6j8q\" (UID: \"f5e9cbc1-01a5-4d28-8287-f100f26a76af\") " pod="openshift-ingress/kube-auth-proxy-864788f75f-x6j8q"
Sep 19 11:38:38.483404 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.483302    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5e9cbc1-01a5-4d28-8287-f100f26a76af-tls-certs\") pod \"kube-auth-proxy-864788f75f-x6j8q\" (UID: \"f5e9cbc1-01a5-4d28-8287-f100f26a76af\") " pod="openshift-ingress/kube-auth-proxy-864788f75f-x6j8q"
Sep 19 11:38:38.565880 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.565852    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:38.584113 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.584071    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb4pt\" (UniqueName: \"kubernetes.io/projected/f5e9cbc1-01a5-4d28-8287-f100f26a76af-kube-api-access-lb4pt\") pod \"kube-auth-proxy-864788f75f-x6j8q\" (UID: \"f5e9cbc1-01a5-4d28-8287-f100f26a76af\") " pod="openshift-ingress/kube-auth-proxy-864788f75f-x6j8q"
Sep 19 11:38:38.584607 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.584133    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5e9cbc1-01a5-4d28-8287-f100f26a76af-tmp\") pod \"kube-auth-proxy-864788f75f-x6j8q\" (UID: \"f5e9cbc1-01a5-4d28-8287-f100f26a76af\") " pod="openshift-ingress/kube-auth-proxy-864788f75f-x6j8q"
Sep 19 11:38:38.584607 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.584157    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5e9cbc1-01a5-4d28-8287-f100f26a76af-tls-certs\") pod \"kube-auth-proxy-864788f75f-x6j8q\" (UID: \"f5e9cbc1-01a5-4d28-8287-f100f26a76af\") " pod="openshift-ingress/kube-auth-proxy-864788f75f-x6j8q"
Sep 19 11:38:38.586783 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.586755    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5e9cbc1-01a5-4d28-8287-f100f26a76af-tmp\") pod \"kube-auth-proxy-864788f75f-x6j8q\" (UID: \"f5e9cbc1-01a5-4d28-8287-f100f26a76af\") " pod="openshift-ingress/kube-auth-proxy-864788f75f-x6j8q"
Sep 19 11:38:38.586965 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.586946    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5e9cbc1-01a5-4d28-8287-f100f26a76af-tls-certs\") pod \"kube-auth-proxy-864788f75f-x6j8q\" (UID: \"f5e9cbc1-01a5-4d28-8287-f100f26a76af\") " pod="openshift-ingress/kube-auth-proxy-864788f75f-x6j8q"
Sep 19 11:38:38.592046 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.591983    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb4pt\" (UniqueName: \"kubernetes.io/projected/f5e9cbc1-01a5-4d28-8287-f100f26a76af-kube-api-access-lb4pt\") pod \"kube-auth-proxy-864788f75f-x6j8q\" (UID: \"f5e9cbc1-01a5-4d28-8287-f100f26a76af\") " pod="openshift-ingress/kube-auth-proxy-864788f75f-x6j8q"
Sep 19 11:38:38.622084 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.621939    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-k668c"]
Sep 19 11:38:38.624709 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:38:38.624680    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fad1905_a4c8_4520_b0b9_2d1db1fd6624.slice/crio-7103209984c2b3908a42bba43cdbee5d724f097c0db289b8c6ff859864331796 WatchSource:0}: Error finding container 7103209984c2b3908a42bba43cdbee5d724f097c0db289b8c6ff859864331796: Status 404 returned error can't find the container with id 7103209984c2b3908a42bba43cdbee5d724f097c0db289b8c6ff859864331796
Sep 19 11:38:38.667583 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.667547    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-864788f75f-x6j8q"
Sep 19 11:38:38.787257 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.787222    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:38.946050 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.946020    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-864788f75f-x6j8q"]
Sep 19 11:38:38.948554 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:38:38.948525    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e9cbc1_01a5_4d28_8287_f100f26a76af.slice/crio-29e787bcd88f531af9ce149764ca03a5fcd4676c61f1d868a37c47a3068d931a WatchSource:0}: Error finding container 29e787bcd88f531af9ce149764ca03a5fcd4676c61f1d868a37c47a3068d931a: Status 404 returned error can't find the container with id 29e787bcd88f531af9ce149764ca03a5fcd4676c61f1d868a37c47a3068d931a
Sep 19 11:38:38.950558 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:38.950537    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 11:38:39.381989 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:39.381951    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-k668c" event={"ID":"9fad1905-a4c8-4520-b0b9-2d1db1fd6624","Type":"ContainerStarted","Data":"7103209984c2b3908a42bba43cdbee5d724f097c0db289b8c6ff859864331796"}
Sep 19 11:38:39.383285 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:39.383255    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-864788f75f-x6j8q" event={"ID":"f5e9cbc1-01a5-4d28-8287-f100f26a76af","Type":"ContainerStarted","Data":"29e787bcd88f531af9ce149764ca03a5fcd4676c61f1d868a37c47a3068d931a"}
Sep 19 11:38:39.617978 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:39.617933    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:40.142403 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:40.142341    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:40.396507 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:40.396419    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-ks2j6"]
Sep 19 11:38:40.400214 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:40.400186    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-ks2j6"
Sep 19 11:38:40.403098 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:40.403069    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 19 11:38:40.403475 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:40.403454    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 19 11:38:40.404142 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:40.404112    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-z9gwv\""
Sep 19 11:38:40.414141 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:40.413503    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-ks2j6"]
Sep 19 11:38:40.500613 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:40.500563    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj9r8\" (UniqueName: \"kubernetes.io/projected/6eb955af-5023-4977-b253-ce98d2950e19-kube-api-access-fj9r8\") pod \"limitador-operator-controller-manager-6d9b9f85cb-ks2j6\" (UID: \"6eb955af-5023-4977-b253-ce98d2950e19\") " pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-ks2j6"
Sep 19 11:38:40.601437 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:40.601398    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj9r8\" (UniqueName: \"kubernetes.io/projected/6eb955af-5023-4977-b253-ce98d2950e19-kube-api-access-fj9r8\") pod \"limitador-operator-controller-manager-6d9b9f85cb-ks2j6\" (UID: \"6eb955af-5023-4977-b253-ce98d2950e19\") " pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-ks2j6"
Sep 19 11:38:40.617300 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:40.617238    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj9r8\" (UniqueName: \"kubernetes.io/projected/6eb955af-5023-4977-b253-ce98d2950e19-kube-api-access-fj9r8\") pod \"limitador-operator-controller-manager-6d9b9f85cb-ks2j6\" (UID: \"6eb955af-5023-4977-b253-ce98d2950e19\") " pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-ks2j6"
Sep 19 11:38:40.715441 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:40.715350    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-ks2j6"
Sep 19 11:38:41.086699 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:41.086665    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:41.184352 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:41.184316    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:41.869797 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:41.869764    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:42.399049 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:42.398950    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-864788f75f-x6j8q" event={"ID":"f5e9cbc1-01a5-4d28-8287-f100f26a76af","Type":"ContainerStarted","Data":"b848f194ffd52611d422fc63ac0315fc17345fb64223d36e4aba6a858874faab"}
Sep 19 11:38:42.400628 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:42.400589    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-k668c" event={"ID":"9fad1905-a4c8-4520-b0b9-2d1db1fd6624","Type":"ContainerStarted","Data":"41d56e5b6ef2baf8e32c5b3f074fe260431673bf66e4e12a659db65ca7cd9b13"}
Sep 19 11:38:42.400760 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:42.400729    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-k668c"
Sep 19 11:38:42.415951 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:42.415880    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-864788f75f-x6j8q" podStartSLOduration=1.210080748 podStartE2EDuration="4.415864848s" totalImagesPullingTime="3.2057841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:38:42.414622538 +0000 UTC m=+306.614652341" watchObservedRunningTime="2026-09-19 11:38:42.415864848 +0000 UTC m=+306.615894650"
Sep 19 11:38:42.428744 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:42.428698    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-ks2j6"]
Sep 19 11:38:42.431836 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:38:42.431808    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb955af_5023_4977_b253_ce98d2950e19.slice/crio-361c0e59511aa61e0d612b0cfb957399bcf0c5f9823c1832ffc272cf11580783 WatchSource:0}: Error finding container 361c0e59511aa61e0d612b0cfb957399bcf0c5f9823c1832ffc272cf11580783: Status 404 returned error can't find the container with id 361c0e59511aa61e0d612b0cfb957399bcf0c5f9823c1832ffc272cf11580783
Sep 19 11:38:42.440608 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:42.440560    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-k668c" podStartSLOduration=0.904914813 podStartE2EDuration="4.440544234s" totalImagesPullingTime="3.535629421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:38:42.439127133 +0000 UTC m=+306.639156936" watchObservedRunningTime="2026-09-19 11:38:42.440544234 +0000 UTC m=+306.640574038"
Sep 19 11:38:43.306065 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:43.306037    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:43.406522 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:43.406478    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-ks2j6" event={"ID":"6eb955af-5023-4977-b253-ce98d2950e19","Type":"ContainerStarted","Data":"361c0e59511aa61e0d612b0cfb957399bcf0c5f9823c1832ffc272cf11580783"}
Sep 19 11:38:45.418463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:45.418384    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-ks2j6" event={"ID":"6eb955af-5023-4977-b253-ce98d2950e19","Type":"ContainerStarted","Data":"c17ccd155431a0042d09201bffdc585609ef0878eeda503ceb2a86f426bf7c43"}
Sep 19 11:38:45.418854 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:45.418496    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-ks2j6"
Sep 19 11:38:45.440201 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:45.440144    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-ks2j6" podStartSLOduration=2.707456176 podStartE2EDuration="5.440125366s" totalImagesPullingTime="2.73266919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:38:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:38:45.438906881 +0000 UTC m=+309.638936697" watchObservedRunningTime="2026-09-19 11:38:45.440125366 +0000 UTC m=+309.640155179"
Sep 19 11:38:45.677907 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:45.677827    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:46.585840 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:46.585803    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:46.689588 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:46.689553    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:47.482085 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:47.482049    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b449b7679-7bz44"]
Sep 19 11:38:47.485212 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:47.485195    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-7bz44"
Sep 19 11:38:47.487788 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:47.487762    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-dgzkx\""
Sep 19 11:38:47.487788 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:47.487778    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 19 11:38:47.494074 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:47.494051    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b449b7679-7bz44"]
Sep 19 11:38:47.562795 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:47.562759    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtczt\" (UniqueName: \"kubernetes.io/projected/7828b099-93eb-4f9f-84c0-59b1f6c52000-kube-api-access-vtczt\") pod \"dns-operator-controller-manager-7b449b7679-7bz44\" (UID: \"7828b099-93eb-4f9f-84c0-59b1f6c52000\") " pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-7bz44"
Sep 19 11:38:47.663527 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:47.663492    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtczt\" (UniqueName: \"kubernetes.io/projected/7828b099-93eb-4f9f-84c0-59b1f6c52000-kube-api-access-vtczt\") pod \"dns-operator-controller-manager-7b449b7679-7bz44\" (UID: \"7828b099-93eb-4f9f-84c0-59b1f6c52000\") " pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-7bz44"
Sep 19 11:38:47.673301 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:47.673268    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtczt\" (UniqueName: \"kubernetes.io/projected/7828b099-93eb-4f9f-84c0-59b1f6c52000-kube-api-access-vtczt\") pod \"dns-operator-controller-manager-7b449b7679-7bz44\" (UID: \"7828b099-93eb-4f9f-84c0-59b1f6c52000\") " pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-7bz44"
Sep 19 11:38:47.796242 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:47.796150    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-7bz44"
Sep 19 11:38:47.897504 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:47.897469    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:48.064501 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:48.064471    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b449b7679-7bz44"]
Sep 19 11:38:48.067497 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:38:48.067465    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7828b099_93eb_4f9f_84c0_59b1f6c52000.slice/crio-925816670ba37e08e72bfde79cc796dfc09224436d55cc4ad2bc5527ef93f215 WatchSource:0}: Error finding container 925816670ba37e08e72bfde79cc796dfc09224436d55cc4ad2bc5527ef93f215: Status 404 returned error can't find the container with id 925816670ba37e08e72bfde79cc796dfc09224436d55cc4ad2bc5527ef93f215
Sep 19 11:38:48.408547 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:48.408469    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-k668c"
Sep 19 11:38:48.433601 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:48.433565    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-7bz44" event={"ID":"7828b099-93eb-4f9f-84c0-59b1f6c52000","Type":"ContainerStarted","Data":"925816670ba37e08e72bfde79cc796dfc09224436d55cc4ad2bc5527ef93f215"}
Sep 19 11:38:50.445430 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:50.445386    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-7bz44" event={"ID":"7828b099-93eb-4f9f-84c0-59b1f6c52000","Type":"ContainerStarted","Data":"ae28731d9204964c06e24c2cdce80b499f31a72f5ff5b3788c5d58353a91e9ea"}
Sep 19 11:38:50.445861 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:50.445571    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-7bz44"
Sep 19 11:38:50.464195 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:50.464151    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-7bz44" podStartSLOduration=1.994373167 podStartE2EDuration="3.464135776s" totalImagesPullingTime="1.469762609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:38:50.46255325 +0000 UTC m=+314.662583052" watchObservedRunningTime="2026-09-19 11:38:50.464135776 +0000 UTC m=+314.664165578"
Sep 19 11:38:51.267865 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:51.267828    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:51.318282 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:51.318250    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:51.868013 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:51.867982    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:51.919637 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:51.919599    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:52.269958 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:52.269860    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:52.323660 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:52.323630    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:54.366703 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:54.366666    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"]
Sep 19 11:38:54.370015 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:54.369994    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"
Sep 19 11:38:54.373521 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:54.373495    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8nxp4\""
Sep 19 11:38:54.381431 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:54.381405    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"]
Sep 19 11:38:54.520809 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:54.520771    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b233060-ef58-4c7c-a92c-a39ffd617014-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-54b6d5c989-k7fwg\" (UID: \"9b233060-ef58-4c7c-a92c-a39ffd617014\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"
Sep 19 11:38:54.521007 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:54.520823    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw68t\" (UniqueName: \"kubernetes.io/projected/9b233060-ef58-4c7c-a92c-a39ffd617014-kube-api-access-mw68t\") pod \"kuadrant-operator-controller-manager-54b6d5c989-k7fwg\" (UID: \"9b233060-ef58-4c7c-a92c-a39ffd617014\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"
Sep 19 11:38:54.621333 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:54.621246    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b233060-ef58-4c7c-a92c-a39ffd617014-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-54b6d5c989-k7fwg\" (UID: \"9b233060-ef58-4c7c-a92c-a39ffd617014\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"
Sep 19 11:38:54.621333 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:54.621297    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw68t\" (UniqueName: \"kubernetes.io/projected/9b233060-ef58-4c7c-a92c-a39ffd617014-kube-api-access-mw68t\") pod \"kuadrant-operator-controller-manager-54b6d5c989-k7fwg\" (UID: \"9b233060-ef58-4c7c-a92c-a39ffd617014\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"
Sep 19 11:38:54.621636 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:54.621612    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b233060-ef58-4c7c-a92c-a39ffd617014-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-54b6d5c989-k7fwg\" (UID: \"9b233060-ef58-4c7c-a92c-a39ffd617014\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"
Sep 19 11:38:54.630738 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:54.630713    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw68t\" (UniqueName: \"kubernetes.io/projected/9b233060-ef58-4c7c-a92c-a39ffd617014-kube-api-access-mw68t\") pod \"kuadrant-operator-controller-manager-54b6d5c989-k7fwg\" (UID: \"9b233060-ef58-4c7c-a92c-a39ffd617014\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"
Sep 19 11:38:54.681383 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:54.681345    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"
Sep 19 11:38:54.957175 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:54.957148    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"]
Sep 19 11:38:54.959864 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:38:54.959837    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b233060_ef58_4c7c_a92c_a39ffd617014.slice/crio-689aef5bdd195b03779180294681058486d43bdeed48075ac5d38c3cbf8d643b WatchSource:0}: Error finding container 689aef5bdd195b03779180294681058486d43bdeed48075ac5d38c3cbf8d643b: Status 404 returned error can't find the container with id 689aef5bdd195b03779180294681058486d43bdeed48075ac5d38c3cbf8d643b
Sep 19 11:38:55.472437 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:55.472400    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg" event={"ID":"9b233060-ef58-4c7c-a92c-a39ffd617014","Type":"ContainerStarted","Data":"689aef5bdd195b03779180294681058486d43bdeed48075ac5d38c3cbf8d643b"}
Sep 19 11:38:56.426084 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:56.426054    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6d9b9f85cb-ks2j6"
Sep 19 11:38:58.727729 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:58.727693    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:58.830119 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:58.830084    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:38:59.497648 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:59.497610    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg" event={"ID":"9b233060-ef58-4c7c-a92c-a39ffd617014","Type":"ContainerStarted","Data":"a5938c0b26a0a00808b5f4cab92caa38017a11e3c886cba25311e42acfe5d592"}
Sep 19 11:38:59.497883 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:59.497733    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"
Sep 19 11:38:59.520659 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:38:59.520603    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg" podStartSLOduration=1.885215163 podStartE2EDuration="5.520584835s" totalImagesPullingTime="3.635369672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:38:59.517650745 +0000 UTC m=+323.717680549" watchObservedRunningTime="2026-09-19 11:38:59.520584835 +0000 UTC m=+323.720614637"
Sep 19 11:39:01.452621 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:01.452592    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b449b7679-7bz44"
Sep 19 11:39:03.478586 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:03.478548    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:03.578175 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:03.578136    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:05.925745 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:05.925691    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:05.982571 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:05.982537    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:07.945530 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:07.945497    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:10.504580 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:10.504551    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"
Sep 19 11:39:12.473797 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:12.473763    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:12.567253 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:12.567225    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:12.667292 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:12.667261    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:19.475212 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:19.475175    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:19.577551 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:19.577518    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:24.283256 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:24.283201    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:24.382234 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:24.382190    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:26.751948 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:26.751887    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:26.801491 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:26.801459    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:30.167807 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:30.167771    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:30.332526 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:30.332491    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:30.444899 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:30.444809    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:30.913338 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:30.913304    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:31.275584 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.275488    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:31.376630 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.376596    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:31.472484 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.472444    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:31.573614 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.573575    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:31.625942 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.625872    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-7645b9c6f5-cn92z"]
Sep 19 11:39:31.628861 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.628832    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-7645b9c6f5-cn92z"
Sep 19 11:39:31.632258 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.632230    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rgz2z\""
Sep 19 11:39:31.633474 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.633445    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 19 11:39:31.633474 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.633469    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 19 11:39:31.633665 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.633492    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 19 11:39:31.643094 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.643056    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-7645b9c6f5-cn92z"]
Sep 19 11:39:31.661533 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.661503    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:31.671939 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.671875    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-77df98bf98-n72lf"]
Sep 19 11:39:31.674152 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.674133    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-77df98bf98-n72lf"
Sep 19 11:39:31.677377 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.677353    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-hxczq\""
Sep 19 11:39:31.677628 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.677612    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 19 11:39:31.687174 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.687112    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-77df98bf98-n72lf"]
Sep 19 11:39:31.735176 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.735136    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dce098a-bdf6-4503-a8c2-d0d82b0911dd-tls-certs\") pod \"model-serving-api-77df98bf98-n72lf\" (UID: \"0dce098a-bdf6-4503-a8c2-d0d82b0911dd\") " pod="opendatahub/model-serving-api-77df98bf98-n72lf"
Sep 19 11:39:31.735356 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.735203    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xqpr\" (UniqueName: \"kubernetes.io/projected/0dce098a-bdf6-4503-a8c2-d0d82b0911dd-kube-api-access-2xqpr\") pod \"model-serving-api-77df98bf98-n72lf\" (UID: \"0dce098a-bdf6-4503-a8c2-d0d82b0911dd\") " pod="opendatahub/model-serving-api-77df98bf98-n72lf"
Sep 19 11:39:31.735356 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.735248    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/820ea15c-9147-496e-86bd-479a0bdb5def-cert\") pod \"kserve-controller-manager-7645b9c6f5-cn92z\" (UID: \"820ea15c-9147-496e-86bd-479a0bdb5def\") " pod="opendatahub/kserve-controller-manager-7645b9c6f5-cn92z"
Sep 19 11:39:31.735356 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.735329    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kbtx\" (UniqueName: \"kubernetes.io/projected/820ea15c-9147-496e-86bd-479a0bdb5def-kube-api-access-5kbtx\") pod \"kserve-controller-manager-7645b9c6f5-cn92z\" (UID: \"820ea15c-9147-496e-86bd-479a0bdb5def\") " pod="opendatahub/kserve-controller-manager-7645b9c6f5-cn92z"
Sep 19 11:39:31.755596 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.755553    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:31.836377 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.836277    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/820ea15c-9147-496e-86bd-479a0bdb5def-cert\") pod \"kserve-controller-manager-7645b9c6f5-cn92z\" (UID: \"820ea15c-9147-496e-86bd-479a0bdb5def\") " pod="opendatahub/kserve-controller-manager-7645b9c6f5-cn92z"
Sep 19 11:39:31.836377 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.836333    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kbtx\" (UniqueName: \"kubernetes.io/projected/820ea15c-9147-496e-86bd-479a0bdb5def-kube-api-access-5kbtx\") pod \"kserve-controller-manager-7645b9c6f5-cn92z\" (UID: \"820ea15c-9147-496e-86bd-479a0bdb5def\") " pod="opendatahub/kserve-controller-manager-7645b9c6f5-cn92z"
Sep 19 11:39:31.836377 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.836373    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dce098a-bdf6-4503-a8c2-d0d82b0911dd-tls-certs\") pod \"model-serving-api-77df98bf98-n72lf\" (UID: \"0dce098a-bdf6-4503-a8c2-d0d82b0911dd\") " pod="opendatahub/model-serving-api-77df98bf98-n72lf"
Sep 19 11:39:31.836662 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.836414    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xqpr\" (UniqueName: \"kubernetes.io/projected/0dce098a-bdf6-4503-a8c2-d0d82b0911dd-kube-api-access-2xqpr\") pod \"model-serving-api-77df98bf98-n72lf\" (UID: \"0dce098a-bdf6-4503-a8c2-d0d82b0911dd\") " pod="opendatahub/model-serving-api-77df98bf98-n72lf"
Sep 19 11:39:31.839076 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.839041    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dce098a-bdf6-4503-a8c2-d0d82b0911dd-tls-certs\") pod \"model-serving-api-77df98bf98-n72lf\" (UID: \"0dce098a-bdf6-4503-a8c2-d0d82b0911dd\") " pod="opendatahub/model-serving-api-77df98bf98-n72lf"
Sep 19 11:39:31.839076 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.839078    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/820ea15c-9147-496e-86bd-479a0bdb5def-cert\") pod \"kserve-controller-manager-7645b9c6f5-cn92z\" (UID: \"820ea15c-9147-496e-86bd-479a0bdb5def\") " pod="opendatahub/kserve-controller-manager-7645b9c6f5-cn92z"
Sep 19 11:39:31.846186 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.846121    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:31.849218 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.849187    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xqpr\" (UniqueName: \"kubernetes.io/projected/0dce098a-bdf6-4503-a8c2-d0d82b0911dd-kube-api-access-2xqpr\") pod \"model-serving-api-77df98bf98-n72lf\" (UID: \"0dce098a-bdf6-4503-a8c2-d0d82b0911dd\") " pod="opendatahub/model-serving-api-77df98bf98-n72lf"
Sep 19 11:39:31.849218 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.849199    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kbtx\" (UniqueName: \"kubernetes.io/projected/820ea15c-9147-496e-86bd-479a0bdb5def-kube-api-access-5kbtx\") pod \"kserve-controller-manager-7645b9c6f5-cn92z\" (UID: \"820ea15c-9147-496e-86bd-479a0bdb5def\") " pod="opendatahub/kserve-controller-manager-7645b9c6f5-cn92z"
Sep 19 11:39:31.915272 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.915239    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:31.940403 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.940369    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-7645b9c6f5-cn92z"
Sep 19 11:39:31.986492 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:31.986456    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-77df98bf98-n72lf"
Sep 19 11:39:32.231516 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:32.231479    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-7645b9c6f5-cn92z"]
Sep 19 11:39:32.233747 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:39:32.233710    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820ea15c_9147_496e_86bd_479a0bdb5def.slice/crio-0c96810c4cf4dc861d894a12f2809bd1701171a37b17f59194fb62d7aa4f6c06 WatchSource:0}: Error finding container 0c96810c4cf4dc861d894a12f2809bd1701171a37b17f59194fb62d7aa4f6c06: Status 404 returned error can't find the container with id 0c96810c4cf4dc861d894a12f2809bd1701171a37b17f59194fb62d7aa4f6c06
Sep 19 11:39:32.278538 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:32.278514    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-77df98bf98-n72lf"]
Sep 19 11:39:32.280800 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:39:32.280767    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dce098a_bdf6_4503_a8c2_d0d82b0911dd.slice/crio-d1bddf0b3cfa8f21b127baf8ce4d6a9e774182710350c71c8eaf574e76e84bd1 WatchSource:0}: Error finding container d1bddf0b3cfa8f21b127baf8ce4d6a9e774182710350c71c8eaf574e76e84bd1: Status 404 returned error can't find the container with id d1bddf0b3cfa8f21b127baf8ce4d6a9e774182710350c71c8eaf574e76e84bd1
Sep 19 11:39:32.620096 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:32.620066    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:32.654329 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:32.654297    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-77df98bf98-n72lf" event={"ID":"0dce098a-bdf6-4503-a8c2-d0d82b0911dd","Type":"ContainerStarted","Data":"d1bddf0b3cfa8f21b127baf8ce4d6a9e774182710350c71c8eaf574e76e84bd1"}
Sep 19 11:39:32.655513 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:32.655485    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-7645b9c6f5-cn92z" event={"ID":"820ea15c-9147-496e-86bd-479a0bdb5def","Type":"ContainerStarted","Data":"0c96810c4cf4dc861d894a12f2809bd1701171a37b17f59194fb62d7aa4f6c06"}
Sep 19 11:39:32.725853 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:32.725822    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:32.858190 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:32.858151    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:32.954712 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:32.954575    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:33.692031 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:33.691970    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:33.742630 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:33.742591    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:34.088957 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:34.088890    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:34.189199 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:34.189157    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:34.558339 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:34.558242    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:34.621057 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:34.621015    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:35.675410 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:35.675357    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-77df98bf98-n72lf" event={"ID":"0dce098a-bdf6-4503-a8c2-d0d82b0911dd","Type":"ContainerStarted","Data":"6b4991e7dc4ec3aba992f07e2c0cae1a6fb099223d562dcde1fd65ba0c7b014c"}
Sep 19 11:39:35.675850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:35.675466    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-77df98bf98-n72lf"
Sep 19 11:39:35.676837 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:35.676813    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-7645b9c6f5-cn92z" event={"ID":"820ea15c-9147-496e-86bd-479a0bdb5def","Type":"ContainerStarted","Data":"afff2bcb2566067e456a7cd869b3e3ee14654068e4532f68fe1448af45197c63"}
Sep 19 11:39:35.676994 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:35.676973    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-7645b9c6f5-cn92z"
Sep 19 11:39:35.707502 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:35.707452    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-77df98bf98-n72lf" podStartSLOduration=1.8510540899999999 podStartE2EDuration="4.70743463s" totalImagesPullingTime="2.85638054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:39:35.704881411 +0000 UTC m=+359.904911213" watchObservedRunningTime="2026-09-19 11:39:35.70743463 +0000 UTC m=+359.907464432"
Sep 19 11:39:35.742197 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:35.742144    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-7645b9c6f5-cn92z" podStartSLOduration=1.835515352 podStartE2EDuration="4.742127482s" totalImagesPullingTime="2.90661213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:39:35.740500896 +0000 UTC m=+359.940530699" watchObservedRunningTime="2026-09-19 11:39:35.742127482 +0000 UTC m=+359.942157315"
Sep 19 11:39:38.948607 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:38.948554    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:39.049235 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:39.049199    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:41.493633 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:41.493598    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:41.594384 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:41.594353    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:45.588014 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:45.587985    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:45.718848 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:45.718807    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:45.900518 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:45.900419    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:46.001156 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:46.001122    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:46.105089 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:46.105053    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:46.686788 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:46.686759    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-77df98bf98-n72lf"
Sep 19 11:39:48.344846 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:48.344797    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:48.396949 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:48.396900    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:53.656161 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:53.656120    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:39:53.756517 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:39:53.756484    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:02.928409 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:02.928365    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:03.026736 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:03.026707    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:03.075723 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:03.075694    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:03.242698 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:03.242608    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:03.494139 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:03.494057    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:06.687147 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:06.687117    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-7645b9c6f5-cn92z"
Sep 19 11:40:06.956900 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:06.956812    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:07.060965 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:07.060932    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:16.760931 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:16.760872    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:16.812744 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:16.812709    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:18.650033 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:18.649995    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:18.752113 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:18.752081    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:27.710968 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:27.710927    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:27.761349 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:27.761318    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:27.816497 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:27.816465    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:36.557733 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:36.557696    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:36.656556 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:36.656525    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:38.065453 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.065421    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:38.168730 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.168693    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:38.206230 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.206191    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"]
Sep 19 11:40:38.209347 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.209324    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"
Sep 19 11:40:38.212207 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.212176    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-g4wpl\""
Sep 19 11:40:38.212207 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.212202    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 19 11:40:38.212443 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.212182    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 19 11:40:38.212443 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.212256    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 19 11:40:38.212443 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.212257    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 11:40:38.221263 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.221233    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"]
Sep 19 11:40:38.267234 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.267189    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:38.307484 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.307445    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw8q9\" (UniqueName: \"kubernetes.io/projected/5ffb2c81-22d2-4113-895e-58471105ccbc-kube-api-access-zw8q9\") pod \"maas-api-d9c5bbd4f-kz2md\" (UID: \"5ffb2c81-22d2-4113-895e-58471105ccbc\") " pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"
Sep 19 11:40:38.307676 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.307493    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ffb2c81-22d2-4113-895e-58471105ccbc-maas-api-tls\") pod \"maas-api-d9c5bbd4f-kz2md\" (UID: \"5ffb2c81-22d2-4113-895e-58471105ccbc\") " pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"
Sep 19 11:40:38.307676 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.307576    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffb2c81-22d2-4113-895e-58471105ccbc-openshift-service-ca\") pod \"maas-api-d9c5bbd4f-kz2md\" (UID: \"5ffb2c81-22d2-4113-895e-58471105ccbc\") " pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"
Sep 19 11:40:38.307676 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.307633    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ffb2c81-22d2-4113-895e-58471105ccbc-metrics-certs\") pod \"maas-api-d9c5bbd4f-kz2md\" (UID: \"5ffb2c81-22d2-4113-895e-58471105ccbc\") " pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"
Sep 19 11:40:38.369390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.369295    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:38.408038 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.408009    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ffb2c81-22d2-4113-895e-58471105ccbc-maas-api-tls\") pod \"maas-api-d9c5bbd4f-kz2md\" (UID: \"5ffb2c81-22d2-4113-895e-58471105ccbc\") " pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"
Sep 19 11:40:38.408241 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.408061    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffb2c81-22d2-4113-895e-58471105ccbc-openshift-service-ca\") pod \"maas-api-d9c5bbd4f-kz2md\" (UID: \"5ffb2c81-22d2-4113-895e-58471105ccbc\") " pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"
Sep 19 11:40:38.408241 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.408120    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ffb2c81-22d2-4113-895e-58471105ccbc-metrics-certs\") pod \"maas-api-d9c5bbd4f-kz2md\" (UID: \"5ffb2c81-22d2-4113-895e-58471105ccbc\") " pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"
Sep 19 11:40:38.408241 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.408146    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw8q9\" (UniqueName: \"kubernetes.io/projected/5ffb2c81-22d2-4113-895e-58471105ccbc-kube-api-access-zw8q9\") pod \"maas-api-d9c5bbd4f-kz2md\" (UID: \"5ffb2c81-22d2-4113-895e-58471105ccbc\") " pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"
Sep 19 11:40:38.408792 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.408765    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffb2c81-22d2-4113-895e-58471105ccbc-openshift-service-ca\") pod \"maas-api-d9c5bbd4f-kz2md\" (UID: \"5ffb2c81-22d2-4113-895e-58471105ccbc\") " pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"
Sep 19 11:40:38.410973 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.410897    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ffb2c81-22d2-4113-895e-58471105ccbc-metrics-certs\") pod \"maas-api-d9c5bbd4f-kz2md\" (UID: \"5ffb2c81-22d2-4113-895e-58471105ccbc\") " pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"
Sep 19 11:40:38.411192 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.411099    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ffb2c81-22d2-4113-895e-58471105ccbc-maas-api-tls\") pod \"maas-api-d9c5bbd4f-kz2md\" (UID: \"5ffb2c81-22d2-4113-895e-58471105ccbc\") " pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"
Sep 19 11:40:38.420116 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.420082    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw8q9\" (UniqueName: \"kubernetes.io/projected/5ffb2c81-22d2-4113-895e-58471105ccbc-kube-api-access-zw8q9\") pod \"maas-api-d9c5bbd4f-kz2md\" (UID: \"5ffb2c81-22d2-4113-895e-58471105ccbc\") " pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"
Sep 19 11:40:38.521094 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.521055    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"
Sep 19 11:40:38.554169 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.554131    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:38.652484 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.652396    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:38.800042 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.800005    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"]
Sep 19 11:40:38.803394 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:40:38.803362    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ffb2c81_22d2_4113_895e_58471105ccbc.slice/crio-ac80eb8339f90b797c224a7f55a945e585f5a0603b521340be4e276c9504e79c WatchSource:0}: Error finding container ac80eb8339f90b797c224a7f55a945e585f5a0603b521340be4e276c9504e79c: Status 404 returned error can't find the container with id ac80eb8339f90b797c224a7f55a945e585f5a0603b521340be4e276c9504e79c
Sep 19 11:40:38.983116 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:38.983019    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md" event={"ID":"5ffb2c81-22d2-4113-895e-58471105ccbc","Type":"ContainerStarted","Data":"ac80eb8339f90b797c224a7f55a945e585f5a0603b521340be4e276c9504e79c"}
Sep 19 11:40:40.995851 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:40.995817    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md" event={"ID":"5ffb2c81-22d2-4113-895e-58471105ccbc","Type":"ContainerStarted","Data":"5a09f483672ac341fb8a936f44c4543e2acc2b7752db9aa2a40f7b41ee897421"}
Sep 19 11:40:40.996298 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:40.995953    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"
Sep 19 11:40:41.014015 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:41.013956    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md" podStartSLOduration=1.379181805 podStartE2EDuration="3.013938143s" totalImagesPullingTime="1.634756338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:40:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:40:41.01231727 +0000 UTC m=+425.212347074" watchObservedRunningTime="2026-09-19 11:40:41.013938143 +0000 UTC m=+425.213967945"
Sep 19 11:40:41.337431 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:41.337394    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:41.438611 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:41.438577    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:47.006568 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:47.006538    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"
Sep 19 11:40:47.305314 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:47.305233    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:47.739182 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:47.739101    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:50.624569 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:50.624524    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:50.725544 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:50.725513    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:40:56.767328 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:56.767294    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"]
Sep 19 11:40:56.767839 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:56.767484    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md" podUID="5ffb2c81-22d2-4113-895e-58471105ccbc" containerName="maas-api" containerID="cri-o://5a09f483672ac341fb8a936f44c4543e2acc2b7752db9aa2a40f7b41ee897421" gracePeriod=30
Sep 19 11:40:57.021658 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.021580    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"
Sep 19 11:40:57.074312 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.074276    2584 generic.go:356] "Generic (PLEG): container finished" podID="5ffb2c81-22d2-4113-895e-58471105ccbc" containerID="5a09f483672ac341fb8a936f44c4543e2acc2b7752db9aa2a40f7b41ee897421" exitCode=0
Sep 19 11:40:57.074506 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.074324    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md" event={"ID":"5ffb2c81-22d2-4113-895e-58471105ccbc","Type":"ContainerDied","Data":"5a09f483672ac341fb8a936f44c4543e2acc2b7752db9aa2a40f7b41ee897421"}
Sep 19 11:40:57.074506 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.074370    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md" event={"ID":"5ffb2c81-22d2-4113-895e-58471105ccbc","Type":"ContainerDied","Data":"ac80eb8339f90b797c224a7f55a945e585f5a0603b521340be4e276c9504e79c"}
Sep 19 11:40:57.074506 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.074392    2584 scope.go:117] "RemoveContainer" containerID="5a09f483672ac341fb8a936f44c4543e2acc2b7752db9aa2a40f7b41ee897421"
Sep 19 11:40:57.074506 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.074331    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"
Sep 19 11:40:57.083058 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.083032    2584 scope.go:117] "RemoveContainer" containerID="5a09f483672ac341fb8a936f44c4543e2acc2b7752db9aa2a40f7b41ee897421"
Sep 19 11:40:57.083386 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:40:57.083367    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a09f483672ac341fb8a936f44c4543e2acc2b7752db9aa2a40f7b41ee897421\": container with ID starting with 5a09f483672ac341fb8a936f44c4543e2acc2b7752db9aa2a40f7b41ee897421 not found: ID does not exist" containerID="5a09f483672ac341fb8a936f44c4543e2acc2b7752db9aa2a40f7b41ee897421"
Sep 19 11:40:57.083445 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.083396    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a09f483672ac341fb8a936f44c4543e2acc2b7752db9aa2a40f7b41ee897421"} err="failed to get container status \"5a09f483672ac341fb8a936f44c4543e2acc2b7752db9aa2a40f7b41ee897421\": rpc error: code = NotFound desc = could not find container \"5a09f483672ac341fb8a936f44c4543e2acc2b7752db9aa2a40f7b41ee897421\": container with ID starting with 5a09f483672ac341fb8a936f44c4543e2acc2b7752db9aa2a40f7b41ee897421 not found: ID does not exist"
Sep 19 11:40:57.177891 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.177855    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw8q9\" (UniqueName: \"kubernetes.io/projected/5ffb2c81-22d2-4113-895e-58471105ccbc-kube-api-access-zw8q9\") pod \"5ffb2c81-22d2-4113-895e-58471105ccbc\" (UID: \"5ffb2c81-22d2-4113-895e-58471105ccbc\") "
Sep 19 11:40:57.178150 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.177976    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffb2c81-22d2-4113-895e-58471105ccbc-openshift-service-ca\") pod \"5ffb2c81-22d2-4113-895e-58471105ccbc\" (UID: \"5ffb2c81-22d2-4113-895e-58471105ccbc\") "
Sep 19 11:40:57.178150 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.177999    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ffb2c81-22d2-4113-895e-58471105ccbc-metrics-certs\") pod \"5ffb2c81-22d2-4113-895e-58471105ccbc\" (UID: \"5ffb2c81-22d2-4113-895e-58471105ccbc\") "
Sep 19 11:40:57.178150 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.178075    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ffb2c81-22d2-4113-895e-58471105ccbc-maas-api-tls\") pod \"5ffb2c81-22d2-4113-895e-58471105ccbc\" (UID: \"5ffb2c81-22d2-4113-895e-58471105ccbc\") "
Sep 19 11:40:57.178379 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.178358    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ffb2c81-22d2-4113-895e-58471105ccbc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5ffb2c81-22d2-4113-895e-58471105ccbc" (UID: "5ffb2c81-22d2-4113-895e-58471105ccbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:40:57.180370 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.180342    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ffb2c81-22d2-4113-895e-58471105ccbc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5ffb2c81-22d2-4113-895e-58471105ccbc" (UID: "5ffb2c81-22d2-4113-895e-58471105ccbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:40:57.180370 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.180347    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ffb2c81-22d2-4113-895e-58471105ccbc-kube-api-access-zw8q9" (OuterVolumeSpecName: "kube-api-access-zw8q9") pod "5ffb2c81-22d2-4113-895e-58471105ccbc" (UID: "5ffb2c81-22d2-4113-895e-58471105ccbc"). InnerVolumeSpecName "kube-api-access-zw8q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:40:57.180543 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.180392    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ffb2c81-22d2-4113-895e-58471105ccbc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5ffb2c81-22d2-4113-895e-58471105ccbc" (UID: "5ffb2c81-22d2-4113-895e-58471105ccbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:40:57.279518 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.279430    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffb2c81-22d2-4113-895e-58471105ccbc-openshift-service-ca\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:40:57.279518 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.279463    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ffb2c81-22d2-4113-895e-58471105ccbc-metrics-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:40:57.279518 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.279473    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ffb2c81-22d2-4113-895e-58471105ccbc-maas-api-tls\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:40:57.279518 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.279483    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw8q9\" (UniqueName: \"kubernetes.io/projected/5ffb2c81-22d2-4113-895e-58471105ccbc-kube-api-access-zw8q9\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:40:57.397342 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.397306    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"]
Sep 19 11:40:57.398855 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.398833    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md"]
Sep 19 11:40:57.903533 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:40:57.903498    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ffb2c81-22d2-4113-895e-58471105ccbc" path="/var/lib/kubelet/pods/5ffb2c81-22d2-4113-895e-58471105ccbc/volumes"
Sep 19 11:41:00.001586 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:00.001536    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-d9c5bbd4f-kz2md" podUID="5ffb2c81-22d2-4113-895e-58471105ccbc" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.33:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 19 11:41:03.674249 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:03.674216    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:41:24.224186 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:24.224147    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:41:38.043755 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.043718    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"]
Sep 19 11:41:38.044523 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.044498    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ffb2c81-22d2-4113-895e-58471105ccbc" containerName="maas-api"
Sep 19 11:41:38.044712 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.044681    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ffb2c81-22d2-4113-895e-58471105ccbc" containerName="maas-api"
Sep 19 11:41:38.044906 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.044892    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ffb2c81-22d2-4113-895e-58471105ccbc" containerName="maas-api"
Sep 19 11:41:38.050003 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.049887    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.052700 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.052674    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6gh2s\""
Sep 19 11:41:38.053839 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.053812    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"]
Sep 19 11:41:38.053983 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.053942    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 19 11:41:38.054056 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.054024    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 19 11:41:38.054371 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.054349    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 19 11:41:38.067806 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.067756    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:41:38.140996 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.140878    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d01fa8b2-646f-4958-8b31-46c250d755f5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-698db7d589-zjppm\" (UID: \"d01fa8b2-646f-4958-8b31-46c250d755f5\") " pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.140996 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.140962    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqtf6\" (UniqueName: \"kubernetes.io/projected/d01fa8b2-646f-4958-8b31-46c250d755f5-kube-api-access-xqtf6\") pod \"e2e-embedding-simulated-kserve-698db7d589-zjppm\" (UID: \"d01fa8b2-646f-4958-8b31-46c250d755f5\") " pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.141225 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.141039    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d01fa8b2-646f-4958-8b31-46c250d755f5-home\") pod \"e2e-embedding-simulated-kserve-698db7d589-zjppm\" (UID: \"d01fa8b2-646f-4958-8b31-46c250d755f5\") " pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.141225 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.141080    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d01fa8b2-646f-4958-8b31-46c250d755f5-model-cache\") pod \"e2e-embedding-simulated-kserve-698db7d589-zjppm\" (UID: \"d01fa8b2-646f-4958-8b31-46c250d755f5\") " pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.141225 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.141162    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d01fa8b2-646f-4958-8b31-46c250d755f5-dshm\") pod \"e2e-embedding-simulated-kserve-698db7d589-zjppm\" (UID: \"d01fa8b2-646f-4958-8b31-46c250d755f5\") " pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.141225 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.141203    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d01fa8b2-646f-4958-8b31-46c250d755f5-tls-certs\") pod \"e2e-embedding-simulated-kserve-698db7d589-zjppm\" (UID: \"d01fa8b2-646f-4958-8b31-46c250d755f5\") " pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.242147 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.242101    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqtf6\" (UniqueName: \"kubernetes.io/projected/d01fa8b2-646f-4958-8b31-46c250d755f5-kube-api-access-xqtf6\") pod \"e2e-embedding-simulated-kserve-698db7d589-zjppm\" (UID: \"d01fa8b2-646f-4958-8b31-46c250d755f5\") " pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.242340 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.242164    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d01fa8b2-646f-4958-8b31-46c250d755f5-home\") pod \"e2e-embedding-simulated-kserve-698db7d589-zjppm\" (UID: \"d01fa8b2-646f-4958-8b31-46c250d755f5\") " pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.242340 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.242200    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d01fa8b2-646f-4958-8b31-46c250d755f5-model-cache\") pod \"e2e-embedding-simulated-kserve-698db7d589-zjppm\" (UID: \"d01fa8b2-646f-4958-8b31-46c250d755f5\") " pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.242340 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.242269    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d01fa8b2-646f-4958-8b31-46c250d755f5-dshm\") pod \"e2e-embedding-simulated-kserve-698db7d589-zjppm\" (UID: \"d01fa8b2-646f-4958-8b31-46c250d755f5\") " pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.242340 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.242294    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d01fa8b2-646f-4958-8b31-46c250d755f5-tls-certs\") pod \"e2e-embedding-simulated-kserve-698db7d589-zjppm\" (UID: \"d01fa8b2-646f-4958-8b31-46c250d755f5\") " pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.242340 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.242330    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d01fa8b2-646f-4958-8b31-46c250d755f5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-698db7d589-zjppm\" (UID: \"d01fa8b2-646f-4958-8b31-46c250d755f5\") " pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.242648 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.242623    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d01fa8b2-646f-4958-8b31-46c250d755f5-model-cache\") pod \"e2e-embedding-simulated-kserve-698db7d589-zjppm\" (UID: \"d01fa8b2-646f-4958-8b31-46c250d755f5\") " pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.242727 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.242687    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d01fa8b2-646f-4958-8b31-46c250d755f5-home\") pod \"e2e-embedding-simulated-kserve-698db7d589-zjppm\" (UID: \"d01fa8b2-646f-4958-8b31-46c250d755f5\") " pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.242785 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.242728    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d01fa8b2-646f-4958-8b31-46c250d755f5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-698db7d589-zjppm\" (UID: \"d01fa8b2-646f-4958-8b31-46c250d755f5\") " pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.244719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.244699    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d01fa8b2-646f-4958-8b31-46c250d755f5-dshm\") pod \"e2e-embedding-simulated-kserve-698db7d589-zjppm\" (UID: \"d01fa8b2-646f-4958-8b31-46c250d755f5\") " pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.244889 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.244874    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d01fa8b2-646f-4958-8b31-46c250d755f5-tls-certs\") pod \"e2e-embedding-simulated-kserve-698db7d589-zjppm\" (UID: \"d01fa8b2-646f-4958-8b31-46c250d755f5\") " pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.250704 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.250672    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqtf6\" (UniqueName: \"kubernetes.io/projected/d01fa8b2-646f-4958-8b31-46c250d755f5-kube-api-access-xqtf6\") pod \"e2e-embedding-simulated-kserve-698db7d589-zjppm\" (UID: \"d01fa8b2-646f-4958-8b31-46c250d755f5\") " pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.363598 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.363553    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:38.636899 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:38.636863    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"]
Sep 19 11:41:38.640258 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:41:38.640226    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd01fa8b2_646f_4958_8b31_46c250d755f5.slice/crio-a501d7ec2175d531132506d8df2822ae1f1a8d4da7c72f1df2be118d35dac493 WatchSource:0}: Error finding container a501d7ec2175d531132506d8df2822ae1f1a8d4da7c72f1df2be118d35dac493: Status 404 returned error can't find the container with id a501d7ec2175d531132506d8df2822ae1f1a8d4da7c72f1df2be118d35dac493
Sep 19 11:41:39.280737 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:39.280702    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm" event={"ID":"d01fa8b2-646f-4958-8b31-46c250d755f5","Type":"ContainerStarted","Data":"a501d7ec2175d531132506d8df2822ae1f1a8d4da7c72f1df2be118d35dac493"}
Sep 19 11:41:40.287542 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:40.287493    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm" event={"ID":"d01fa8b2-646f-4958-8b31-46c250d755f5","Type":"ContainerStarted","Data":"2953a8c8a88214a908ad7f2dd7fb39f099158342ff30ab6fd5fcab557e84a02d"}
Sep 19 11:41:40.309864 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:40.309805    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm" podStartSLOduration=1.29474204 podStartE2EDuration="2.309785906s" totalImagesPullingTime="1.015043866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:41:40.307769149 +0000 UTC m=+484.507798952" watchObservedRunningTime="2026-09-19 11:41:40.309785906 +0000 UTC m=+484.509815709"
Sep 19 11:41:40.363929 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:40.363888    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:41.037565 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.037528    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"]
Sep 19 11:41:41.040989 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.040969    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.043553 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.043522    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 19 11:41:41.047795 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.047757    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"]
Sep 19 11:41:41.126614 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.126584    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:41:41.169738 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.169702    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aff31354-fd81-4741-860a-2506c059d365-dshm\") pod \"e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8\" (UID: \"aff31354-fd81-4741-860a-2506c059d365\") " pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.169907 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.169748    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aff31354-fd81-4741-860a-2506c059d365-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8\" (UID: \"aff31354-fd81-4741-860a-2506c059d365\") " pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.169907 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.169769    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aff31354-fd81-4741-860a-2506c059d365-home\") pod \"e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8\" (UID: \"aff31354-fd81-4741-860a-2506c059d365\") " pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.169907 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.169837    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtk2t\" (UniqueName: \"kubernetes.io/projected/aff31354-fd81-4741-860a-2506c059d365-kube-api-access-vtk2t\") pod \"e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8\" (UID: \"aff31354-fd81-4741-860a-2506c059d365\") " pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.169907 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.169887    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aff31354-fd81-4741-860a-2506c059d365-model-cache\") pod \"e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8\" (UID: \"aff31354-fd81-4741-860a-2506c059d365\") " pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.169907 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.169903    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff31354-fd81-4741-860a-2506c059d365-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8\" (UID: \"aff31354-fd81-4741-860a-2506c059d365\") " pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.270431 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.270386    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aff31354-fd81-4741-860a-2506c059d365-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8\" (UID: \"aff31354-fd81-4741-860a-2506c059d365\") " pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.270431 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.270429    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aff31354-fd81-4741-860a-2506c059d365-home\") pod \"e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8\" (UID: \"aff31354-fd81-4741-860a-2506c059d365\") " pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.270694 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.270455    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtk2t\" (UniqueName: \"kubernetes.io/projected/aff31354-fd81-4741-860a-2506c059d365-kube-api-access-vtk2t\") pod \"e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8\" (UID: \"aff31354-fd81-4741-860a-2506c059d365\") " pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.270694 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.270491    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aff31354-fd81-4741-860a-2506c059d365-model-cache\") pod \"e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8\" (UID: \"aff31354-fd81-4741-860a-2506c059d365\") " pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.270694 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.270506    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff31354-fd81-4741-860a-2506c059d365-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8\" (UID: \"aff31354-fd81-4741-860a-2506c059d365\") " pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.270694 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.270551    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aff31354-fd81-4741-860a-2506c059d365-dshm\") pod \"e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8\" (UID: \"aff31354-fd81-4741-860a-2506c059d365\") " pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.270901 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.270876    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aff31354-fd81-4741-860a-2506c059d365-home\") pod \"e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8\" (UID: \"aff31354-fd81-4741-860a-2506c059d365\") " pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.270901 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.270892    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aff31354-fd81-4741-860a-2506c059d365-model-cache\") pod \"e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8\" (UID: \"aff31354-fd81-4741-860a-2506c059d365\") " pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.271037 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.270983    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff31354-fd81-4741-860a-2506c059d365-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8\" (UID: \"aff31354-fd81-4741-860a-2506c059d365\") " pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.273184 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.273152    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aff31354-fd81-4741-860a-2506c059d365-dshm\") pod \"e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8\" (UID: \"aff31354-fd81-4741-860a-2506c059d365\") " pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.273362 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.273343    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aff31354-fd81-4741-860a-2506c059d365-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8\" (UID: \"aff31354-fd81-4741-860a-2506c059d365\") " pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.278708 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.278679    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtk2t\" (UniqueName: \"kubernetes.io/projected/aff31354-fd81-4741-860a-2506c059d365-kube-api-access-vtk2t\") pod \"e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8\" (UID: \"aff31354-fd81-4741-860a-2506c059d365\") " pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.352275 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.352215    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:41.627638 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:41.627612    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"]
Sep 19 11:41:41.630038 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:41:41.630006    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff31354_fd81_4741_860a_2506c059d365.slice/crio-c4f6a54ef7609743056e6267d2d623bb94ddaad4a5e31ba8c44b6ea2036de9a7 WatchSource:0}: Error finding container c4f6a54ef7609743056e6267d2d623bb94ddaad4a5e31ba8c44b6ea2036de9a7: Status 404 returned error can't find the container with id c4f6a54ef7609743056e6267d2d623bb94ddaad4a5e31ba8c44b6ea2036de9a7
Sep 19 11:41:42.301272 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:42.301229    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8" event={"ID":"aff31354-fd81-4741-860a-2506c059d365","Type":"ContainerStarted","Data":"66f1d6d55ebada44fec205076d19a6740f644e884dc6f623cd03ab6967ca1e99"}
Sep 19 11:41:42.301481 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:42.301298    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8" event={"ID":"aff31354-fd81-4741-860a-2506c059d365","Type":"ContainerStarted","Data":"c4f6a54ef7609743056e6267d2d623bb94ddaad4a5e31ba8c44b6ea2036de9a7"}
Sep 19 11:41:42.317575 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:42.317520    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8" podStartSLOduration=1.161942078 podStartE2EDuration="1.317502237s" totalImagesPullingTime="155.560159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:41:42.317127138 +0000 UTC m=+486.517156942" watchObservedRunningTime="2026-09-19 11:41:42.317502237 +0000 UTC m=+486.517532040"
Sep 19 11:41:42.352596 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:42.352547    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:44.457502 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:44.457465    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:41:48.363975 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:48.363935    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:48.377195 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:48.377161    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:49.347723 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:49.347678    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-698db7d589-zjppm"
Sep 19 11:41:51.352781 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:51.352749    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:51.365595 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:51.365565    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:52.363248 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:52.363220    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8"
Sep 19 11:41:59.852734 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:41:59.852697    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:42:13.062976 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:42:13.062936    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:17.887874 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:17.887844    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:17.938055 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:17.938026    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:20.277127 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:20.277088    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:20.377857 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:20.377822    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:21.423787 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:21.423750    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:21.520562 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:21.520532    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:37.859722 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:37.859695    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/1.log"
Sep 19 11:43:37.861226 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:37.861205    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/1.log"
Sep 19 11:43:37.865551 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:37.865532    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6th98_2fed27e0-833f-49a8-b11e-8f61df6e6bd5/ovn-acl-logging/0.log"
Sep 19 11:43:37.866688 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:37.866669    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6th98_2fed27e0-833f-49a8-b11e-8f61df6e6bd5/ovn-acl-logging/0.log"
Sep 19 11:43:38.787725 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:38.787694    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:39.150833 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.150793    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"]
Sep 19 11:43:39.154227 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.154208    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.156849 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.156824    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 19 11:43:39.156984 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.156891    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-37daed-openshift-default-dockercfg-fq82k\""
Sep 19 11:43:39.165169 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.165144    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"]
Sep 19 11:43:39.227879 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.227837    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istiod-ca-cert\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.228206 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.228173    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-credential-socket\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.228356 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.228285    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-token\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.228356 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.228332    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-workload-socket\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.228471 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.228366    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-workload-certs\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.228471 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.228401    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5rcs\" (UniqueName: \"kubernetes.io/projected/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-kube-api-access-g5rcs\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.228567 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.228492    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-data\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.228567 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.228538    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-envoy\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.228646 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.228580    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-podinfo\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.329181 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.329139    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istiod-ca-cert\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.329388 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.329190    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-credential-socket\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.329388 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.329239    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-token\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.329388 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.329268    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-workload-socket\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.329388 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.329296    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-workload-certs\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.329388 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.329325    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5rcs\" (UniqueName: \"kubernetes.io/projected/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-kube-api-access-g5rcs\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.329388 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.329365    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-data\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.329714 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.329398    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-envoy\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.329714 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.329426    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-podinfo\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.329818 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.329787    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-data\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.329818 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.329808    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-workload-certs\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.329953 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.329862    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-workload-socket\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.330005 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.329951    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-credential-socket\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.330288 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.330263    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istiod-ca-cert\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.332084 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.332056    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-envoy\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.332274 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.332256    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-podinfo\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.336643 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.336618    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-token\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.337003 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.336980    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5rcs\" (UniqueName: \"kubernetes.io/projected/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-kube-api-access-g5rcs\") pod \"e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") " pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.465276 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.465194    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:39.741903 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.741784    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"]
Sep 19 11:43:39.744333 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:43:39.744299    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf9df4b_c8e9_4455_9c2e_a7b6164ea088.slice/crio-ba2fa5960802a8507884e84aafd6b77ad49f18062f24f6cde36eacbea128fb14 WatchSource:0}: Error finding container ba2fa5960802a8507884e84aafd6b77ad49f18062f24f6cde36eacbea128fb14: Status 404 returned error can't find the container with id ba2fa5960802a8507884e84aafd6b77ad49f18062f24f6cde36eacbea128fb14
Sep 19 11:43:39.746496 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.746479    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 11:43:39.903937 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:39.903877    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng" event={"ID":"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088","Type":"ContainerStarted","Data":"ba2fa5960802a8507884e84aafd6b77ad49f18062f24f6cde36eacbea128fb14"}
Sep 19 11:43:40.147836 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:40.147807    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:40.247803 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:40.247756    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:40.584858 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:40.584797    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:40.766846 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:40.766810    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6456-47f5f7"]
Sep 19 11:43:40.770840 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:40.770813    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6456-47f5f7"
Sep 19 11:43:40.773547 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:40.773526    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qhqqh\""
Sep 19 11:43:40.777240 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:40.777212    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6456-47f5f7"]
Sep 19 11:43:40.843874 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:40.843780    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q9x7\" (UniqueName: \"kubernetes.io/projected/89457412-d902-4632-ba3e-4f6357e7338b-kube-api-access-8q9x7\") pod \"test-curl-6456-47f5f7\" (UID: \"89457412-d902-4632-ba3e-4f6357e7338b\") " pod="opendatahub/test-curl-6456-47f5f7"
Sep 19 11:43:40.945048 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:40.944991    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q9x7\" (UniqueName: \"kubernetes.io/projected/89457412-d902-4632-ba3e-4f6357e7338b-kube-api-access-8q9x7\") pod \"test-curl-6456-47f5f7\" (UID: \"89457412-d902-4632-ba3e-4f6357e7338b\") " pod="opendatahub/test-curl-6456-47f5f7"
Sep 19 11:43:40.954521 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:40.954492    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q9x7\" (UniqueName: \"kubernetes.io/projected/89457412-d902-4632-ba3e-4f6357e7338b-kube-api-access-8q9x7\") pod \"test-curl-6456-47f5f7\" (UID: \"89457412-d902-4632-ba3e-4f6357e7338b\") " pod="opendatahub/test-curl-6456-47f5f7"
Sep 19 11:43:41.084252 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:41.084212    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6456-47f5f7"
Sep 19 11:43:41.367653 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:41.367605    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6456-47f5f7"]
Sep 19 11:43:42.340629 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:43:42.340595    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89457412_d902_4632_ba3e_4f6357e7338b.slice/crio-0fc40f42ad778e7c0e6e6f76e1dccf96200477405c35424cacfc9840c92db5e6 WatchSource:0}: Error finding container 0fc40f42ad778e7c0e6e6f76e1dccf96200477405c35424cacfc9840c92db5e6: Status 404 returned error can't find the container with id 0fc40f42ad778e7c0e6e6f76e1dccf96200477405c35424cacfc9840c92db5e6
Sep 19 11:43:42.405261 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:42.405221    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:43:42.405575 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:42.405302    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:43:42.405575 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:42.405332    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:43:42.523242 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:42.523200    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:42.627936 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:42.627836    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:42.926448 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:42.926358    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng" event={"ID":"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088","Type":"ContainerStarted","Data":"9578c3594ebf5961bb2ff5b975165155e314f25f3817200aefa5bcf5a2392ab7"}
Sep 19 11:43:42.927845 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:42.927807    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6456-47f5f7" event={"ID":"89457412-d902-4632-ba3e-4f6357e7338b","Type":"ContainerStarted","Data":"0fc40f42ad778e7c0e6e6f76e1dccf96200477405c35424cacfc9840c92db5e6"}
Sep 19 11:43:42.954511 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:42.954455    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng" podStartSLOduration=1.2962221 podStartE2EDuration="3.954436897s" totalImagesPullingTime="2.658214797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:43:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:43:42.95284603 +0000 UTC m=+607.152875834" watchObservedRunningTime="2026-09-19 11:43:42.954436897 +0000 UTC m=+607.154466700"
Sep 19 11:43:43.465736 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:43.465701    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:43.472347 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:43.472312    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:43.933274 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:43.933239    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:43.934368 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:43.934348    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:43:44.273154 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.273068    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"]
Sep 19 11:43:44.277236 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.277203    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"
Sep 19 11:43:44.279879 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.279854    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 11:43:44.279879 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.279861    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 19 11:43:44.280104 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.279989    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-333f49-serving-cert\""
Sep 19 11:43:44.280104 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.280022    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 19 11:43:44.280216 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.280149    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-g4wpl\""
Sep 19 11:43:44.284171 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.284136    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"]
Sep 19 11:43:44.307401 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.307368    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:44.375658 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.375619    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr7sq\" (UniqueName: \"kubernetes.io/projected/5736df77-9ee0-47f7-b478-00e7a1f0f52d-kube-api-access-zr7sq\") pod \"maas-api-e2e-inf-a-333f49-86869464f5-5l9kw\" (UID: \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"
Sep 19 11:43:44.375847 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.375668    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5736df77-9ee0-47f7-b478-00e7a1f0f52d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-333f49-86869464f5-5l9kw\" (UID: \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"
Sep 19 11:43:44.375847 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.375739    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5736df77-9ee0-47f7-b478-00e7a1f0f52d-maas-api-tls\") pod \"maas-api-e2e-inf-a-333f49-86869464f5-5l9kw\" (UID: \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"
Sep 19 11:43:44.375847 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.375793    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5736df77-9ee0-47f7-b478-00e7a1f0f52d-metrics-certs\") pod \"maas-api-e2e-inf-a-333f49-86869464f5-5l9kw\" (UID: \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"
Sep 19 11:43:44.412412 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.412376    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:44.476718 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.476680    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5736df77-9ee0-47f7-b478-00e7a1f0f52d-maas-api-tls\") pod \"maas-api-e2e-inf-a-333f49-86869464f5-5l9kw\" (UID: \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"
Sep 19 11:43:44.476718 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.476718    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5736df77-9ee0-47f7-b478-00e7a1f0f52d-metrics-certs\") pod \"maas-api-e2e-inf-a-333f49-86869464f5-5l9kw\" (UID: \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"
Sep 19 11:43:44.477233 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.476798    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr7sq\" (UniqueName: \"kubernetes.io/projected/5736df77-9ee0-47f7-b478-00e7a1f0f52d-kube-api-access-zr7sq\") pod \"maas-api-e2e-inf-a-333f49-86869464f5-5l9kw\" (UID: \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"
Sep 19 11:43:44.477233 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.476824    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5736df77-9ee0-47f7-b478-00e7a1f0f52d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-333f49-86869464f5-5l9kw\" (UID: \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"
Sep 19 11:43:44.477985 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.477945    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5736df77-9ee0-47f7-b478-00e7a1f0f52d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-333f49-86869464f5-5l9kw\" (UID: \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"
Sep 19 11:43:44.479602 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.479576    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5736df77-9ee0-47f7-b478-00e7a1f0f52d-metrics-certs\") pod \"maas-api-e2e-inf-a-333f49-86869464f5-5l9kw\" (UID: \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"
Sep 19 11:43:44.479732 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.479681    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5736df77-9ee0-47f7-b478-00e7a1f0f52d-maas-api-tls\") pod \"maas-api-e2e-inf-a-333f49-86869464f5-5l9kw\" (UID: \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"
Sep 19 11:43:44.485327 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.485302    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr7sq\" (UniqueName: \"kubernetes.io/projected/5736df77-9ee0-47f7-b478-00e7a1f0f52d-kube-api-access-zr7sq\") pod \"maas-api-e2e-inf-a-333f49-86869464f5-5l9kw\" (UID: \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"
Sep 19 11:43:44.498377 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.498329    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:44.591800 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.591761    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"
Sep 19 11:43:44.888160 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.888134    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"]
Sep 19 11:43:44.890340 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:43:44.890306    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5736df77_9ee0_47f7_b478_00e7a1f0f52d.slice/crio-aedd62a3e9c5292638d48eb1bfc21b09c088d04c8c88ebfa0b4f1c62266cdf08 WatchSource:0}: Error finding container aedd62a3e9c5292638d48eb1bfc21b09c088d04c8c88ebfa0b4f1c62266cdf08: Status 404 returned error can't find the container with id aedd62a3e9c5292638d48eb1bfc21b09c088d04c8c88ebfa0b4f1c62266cdf08
Sep 19 11:43:44.935639 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.935605    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:44.939756 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.939729    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw" event={"ID":"5736df77-9ee0-47f7-b478-00e7a1f0f52d","Type":"ContainerStarted","Data":"aedd62a3e9c5292638d48eb1bfc21b09c088d04c8c88ebfa0b4f1c62266cdf08"}
Sep 19 11:43:44.941257 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.941232    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6456-47f5f7" event={"ID":"89457412-d902-4632-ba3e-4f6357e7338b","Type":"ContainerStarted","Data":"1a6ea20cd8a541621a9d94d367d79e4f53d53f92ec0344a8397471d9b06b7654"}
Sep 19 11:43:44.973802 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:44.973752    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-6456-47f5f7" podStartSLOduration=3.307072597 podStartE2EDuration="4.973731967s" totalImagesPullingTime="1.66665937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:43:44.971243032 +0000 UTC m=+609.171272833" watchObservedRunningTime="2026-09-19 11:43:44.973731967 +0000 UTC m=+609.173761773"
Sep 19 11:43:45.054509 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:45.054469    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:45.151667 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:45.151589    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:45.536761 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:45.536666    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6456-47f5f7"]
Sep 19 11:43:45.541137 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:45.541105    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6456-47f5f7"]
Sep 19 11:43:45.661935 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:45.661866    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6456-d5d5af"]
Sep 19 11:43:45.662267 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:45.662255    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89457412-d902-4632-ba3e-4f6357e7338b" containerName="test-curl-6456-47f5f7"
Sep 19 11:43:45.662318 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:45.662269    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="89457412-d902-4632-ba3e-4f6357e7338b" containerName="test-curl-6456-47f5f7"
Sep 19 11:43:45.662352 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:45.662332    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="89457412-d902-4632-ba3e-4f6357e7338b" containerName="test-curl-6456-47f5f7"
Sep 19 11:43:45.665372 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:45.665357    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6456-d5d5af"
Sep 19 11:43:45.667708 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:45.667675    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:45.670253 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:45.670228    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6456-d5d5af"]
Sep 19 11:43:45.689173 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:45.689135    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hclk\" (UniqueName: \"kubernetes.io/projected/037662b6-e632-4c81-8894-f6f14a1ffe01-kube-api-access-4hclk\") pod \"test-curl-6456-d5d5af\" (UID: \"037662b6-e632-4c81-8894-f6f14a1ffe01\") " pod="opendatahub/test-curl-6456-d5d5af"
Sep 19 11:43:45.789898 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:45.789804    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hclk\" (UniqueName: \"kubernetes.io/projected/037662b6-e632-4c81-8894-f6f14a1ffe01-kube-api-access-4hclk\") pod \"test-curl-6456-d5d5af\" (UID: \"037662b6-e632-4c81-8894-f6f14a1ffe01\") " pod="opendatahub/test-curl-6456-d5d5af"
Sep 19 11:43:45.798023 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:45.797997    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hclk\" (UniqueName: \"kubernetes.io/projected/037662b6-e632-4c81-8894-f6f14a1ffe01-kube-api-access-4hclk\") pod \"test-curl-6456-d5d5af\" (UID: \"037662b6-e632-4c81-8894-f6f14a1ffe01\") " pod="opendatahub/test-curl-6456-d5d5af"
Sep 19 11:43:45.947175 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:45.947138    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw" event={"ID":"5736df77-9ee0-47f7-b478-00e7a1f0f52d","Type":"ContainerStarted","Data":"e187e452e6d5b2dd71d18c44a4b9883402cc790d3dfe1e58c72df55b42b5363d"}
Sep 19 11:43:45.947500 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:43:45.947479    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-6456-47f5f7" container="test-curl-6456-47f5f7"
Sep 19 11:43:45.966864 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:45.966805    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw" podStartSLOduration=1.582228163 podStartE2EDuration="1.966785817s" totalImagesPullingTime="384.557654ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:43:45.965292264 +0000 UTC m=+610.165322068" watchObservedRunningTime="2026-09-19 11:43:45.966785817 +0000 UTC m=+610.166815619"
Sep 19 11:43:45.976477 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:45.976449    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6456-d5d5af"
Sep 19 11:43:46.247740 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.247706    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6456-d5d5af"]
Sep 19 11:43:46.250303 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:43:46.250277    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod037662b6_e632_4c81_8894_f6f14a1ffe01.slice/crio-6a4590daab608a87179eaebedf3310101bebed09001b13a30724ff10b041031f WatchSource:0}: Error finding container 6a4590daab608a87179eaebedf3310101bebed09001b13a30724ff10b041031f: Status 404 returned error can't find the container with id 6a4590daab608a87179eaebedf3310101bebed09001b13a30724ff10b041031f
Sep 19 11:43:46.428837 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.428801    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"]
Sep 19 11:43:46.432508 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.432483    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"
Sep 19 11:43:46.435019 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.434986    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-7eb15e-serving-cert\""
Sep 19 11:43:46.444098 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.444021    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"]
Sep 19 11:43:46.463753 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.463716    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:46.497244 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.497157    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-metrics-certs\") pod \"maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl\" (UID: \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"
Sep 19 11:43:46.497244 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.497201    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssmvz\" (UniqueName: \"kubernetes.io/projected/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-kube-api-access-ssmvz\") pod \"maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl\" (UID: \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"
Sep 19 11:43:46.497244 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.497233    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl\" (UID: \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"
Sep 19 11:43:46.497453 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.497284    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl\" (UID: \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"
Sep 19 11:43:46.525447 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.525409    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:46.598622 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.598574    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl\" (UID: \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"
Sep 19 11:43:46.599103 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.598655    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-metrics-certs\") pod \"maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl\" (UID: \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"
Sep 19 11:43:46.599103 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.598689    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssmvz\" (UniqueName: \"kubernetes.io/projected/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-kube-api-access-ssmvz\") pod \"maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl\" (UID: \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"
Sep 19 11:43:46.599103 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.598726    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl\" (UID: \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"
Sep 19 11:43:46.599103 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:43:46.598813    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-serving-cert: secret "maas-api-e2e-worker-w0-7eb15e-serving-cert" not found
Sep 19 11:43:46.599103 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:43:46.598886    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-maas-api-tls podName:64ffcacc-383a-4bf4-bb7e-1e03ebafdf65 nodeName:}" failed. No retries permitted until 2026-09-19 11:43:47.098866744 +0000 UTC m=+611.298896526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-maas-api-tls") pod "maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl" (UID: "64ffcacc-383a-4bf4-bb7e-1e03ebafdf65") : secret "maas-api-e2e-worker-w0-7eb15e-serving-cert" not found
Sep 19 11:43:46.599380 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.599289    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl\" (UID: \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"
Sep 19 11:43:46.601430 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.601406    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-metrics-certs\") pod \"maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl\" (UID: \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"
Sep 19 11:43:46.607490 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.607460    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssmvz\" (UniqueName: \"kubernetes.io/projected/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-kube-api-access-ssmvz\") pod \"maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl\" (UID: \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"
Sep 19 11:43:46.953832 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.953794    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6456-d5d5af" event={"ID":"037662b6-e632-4c81-8894-f6f14a1ffe01","Type":"ContainerStarted","Data":"845237a22738816c7a8dd98adbc5a7c2faa25c7b1e80a9e32c9f9e69af13602e"}
Sep 19 11:43:46.954078 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.953841    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6456-d5d5af" event={"ID":"037662b6-e632-4c81-8894-f6f14a1ffe01","Type":"ContainerStarted","Data":"6a4590daab608a87179eaebedf3310101bebed09001b13a30724ff10b041031f"}
Sep 19 11:43:46.954078 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.953944    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-6456-47f5f7" podUID="89457412-d902-4632-ba3e-4f6357e7338b" containerName="test-curl-6456-47f5f7" containerID="cri-o://1a6ea20cd8a541621a9d94d367d79e4f53d53f92ec0344a8397471d9b06b7654" gracePeriod=2
Sep 19 11:43:46.954234 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.954091    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"
Sep 19 11:43:46.975214 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:46.975162    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-6456-d5d5af" podStartSLOduration=1.975144233 podStartE2EDuration="1.975144233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:43:46.973323809 +0000 UTC m=+611.173353615" watchObservedRunningTime="2026-09-19 11:43:46.975144233 +0000 UTC m=+611.175174036"
Sep 19 11:43:47.040429 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:47.040396    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:47.103263 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:47.103225    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl\" (UID: \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"
Sep 19 11:43:47.105984 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:47.105955    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl\" (UID: \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"
Sep 19 11:43:47.140615 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:47.140579    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:47.300899 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:47.300782    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:47.345013 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:47.344975    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"
Sep 19 11:43:47.402185 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:47.401894    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:47.501715 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:47.501676    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:47.547153 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:47.547094    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6456-d5d5af"]
Sep 19 11:43:47.549472 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:47.549444    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6456-d5d5af"]
Sep 19 11:43:47.602041 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:47.602013    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:47.624395 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:47.624368    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"]
Sep 19 11:43:47.627149 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:43:47.627110    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ffcacc_383a_4bf4_bb7e_1e03ebafdf65.slice/crio-95924e938865fa3b6f57a50e3f501e78b98bda22f3f34725c3620b0483f1d7f9 WatchSource:0}: Error finding container 95924e938865fa3b6f57a50e3f501e78b98bda22f3f34725c3620b0483f1d7f9: Status 404 returned error can't find the container with id 95924e938865fa3b6f57a50e3f501e78b98bda22f3f34725c3620b0483f1d7f9
Sep 19 11:43:47.830098 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:47.830060    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:47.959783 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:47.959691    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl" event={"ID":"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65","Type":"ContainerStarted","Data":"95924e938865fa3b6f57a50e3f501e78b98bda22f3f34725c3620b0483f1d7f9"}
Sep 19 11:43:47.960125 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:43:47.960036    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-6456-d5d5af" container="test-curl-6456-d5d5af"
Sep 19 11:43:48.033977 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.033941    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:48.131828 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.131789    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:48.381782 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.381742    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:48.437691 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.437659    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"]
Sep 19 11:43:48.438071 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.438058    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="037662b6-e632-4c81-8894-f6f14a1ffe01" containerName="test-curl-6456-d5d5af"
Sep 19 11:43:48.438127 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.438072    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="037662b6-e632-4c81-8894-f6f14a1ffe01" containerName="test-curl-6456-d5d5af"
Sep 19 11:43:48.438163 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.438157    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="037662b6-e632-4c81-8894-f6f14a1ffe01" containerName="test-curl-6456-d5d5af"
Sep 19 11:43:48.441149 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.441131    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"
Sep 19 11:43:48.443889 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.443867    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-37daed-serving-cert\""
Sep 19 11:43:48.452969 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.452944    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"]
Sep 19 11:43:48.473845 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.473743    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:48.516088 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.516052    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba69ec7a-0643-4b63-9f60-093bd4a10f26-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f\" (UID: \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"
Sep 19 11:43:48.516303 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.516097    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba69ec7a-0643-4b63-9f60-093bd4a10f26-metrics-certs\") pod \"maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f\" (UID: \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"
Sep 19 11:43:48.516303 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.516253    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtzxd\" (UniqueName: \"kubernetes.io/projected/ba69ec7a-0643-4b63-9f60-093bd4a10f26-kube-api-access-gtzxd\") pod \"maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f\" (UID: \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"
Sep 19 11:43:48.516303 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.516295    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba69ec7a-0643-4b63-9f60-093bd4a10f26-maas-api-tls\") pod \"maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f\" (UID: \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"
Sep 19 11:43:48.535351 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.535314    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:48.617516 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.617475    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtzxd\" (UniqueName: \"kubernetes.io/projected/ba69ec7a-0643-4b63-9f60-093bd4a10f26-kube-api-access-gtzxd\") pod \"maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f\" (UID: \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"
Sep 19 11:43:48.618029 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.617524    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba69ec7a-0643-4b63-9f60-093bd4a10f26-maas-api-tls\") pod \"maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f\" (UID: \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"
Sep 19 11:43:48.618029 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.617603    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba69ec7a-0643-4b63-9f60-093bd4a10f26-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f\" (UID: \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"
Sep 19 11:43:48.618029 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.617648    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba69ec7a-0643-4b63-9f60-093bd4a10f26-metrics-certs\") pod \"maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f\" (UID: \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"
Sep 19 11:43:48.618029 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:43:48.617720    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-serving-cert: secret "maas-api-e2e-worker-w4-37daed-serving-cert" not found
Sep 19 11:43:48.618029 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:43:48.617794    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba69ec7a-0643-4b63-9f60-093bd4a10f26-maas-api-tls podName:ba69ec7a-0643-4b63-9f60-093bd4a10f26 nodeName:}" failed. No retries permitted until 2026-09-19 11:43:49.117771796 +0000 UTC m=+613.317801583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ba69ec7a-0643-4b63-9f60-093bd4a10f26-maas-api-tls") pod "maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f" (UID: "ba69ec7a-0643-4b63-9f60-093bd4a10f26") : secret "maas-api-e2e-worker-w4-37daed-serving-cert" not found
Sep 19 11:43:48.618570 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.618549    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba69ec7a-0643-4b63-9f60-093bd4a10f26-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f\" (UID: \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"
Sep 19 11:43:48.620489 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.620467    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba69ec7a-0643-4b63-9f60-093bd4a10f26-metrics-certs\") pod \"maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f\" (UID: \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"
Sep 19 11:43:48.626427 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.626398    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtzxd\" (UniqueName: \"kubernetes.io/projected/ba69ec7a-0643-4b63-9f60-093bd4a10f26-kube-api-access-gtzxd\") pod \"maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f\" (UID: \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"
Sep 19 11:43:48.674744 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.674712    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:48.890514 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.890473    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:48.968734 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.968705    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl" event={"ID":"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65","Type":"ContainerStarted","Data":"26bc0f6c3d638227dd5217d01bea399a763d12d93cbb978441367afa29d484e7"}
Sep 19 11:43:48.968855 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.968765    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-6456-d5d5af" podUID="037662b6-e632-4c81-8894-f6f14a1ffe01" containerName="test-curl-6456-d5d5af" containerID="cri-o://845237a22738816c7a8dd98adbc5a7c2faa25c7b1e80a9e32c9f9e69af13602e" gracePeriod=2
Sep 19 11:43:48.986811 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:48.986763    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl" podStartSLOduration=2.465463453 podStartE2EDuration="2.98674661s" totalImagesPullingTime="521.283157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:43:48.986547472 +0000 UTC m=+613.186577276" watchObservedRunningTime="2026-09-19 11:43:48.98674661 +0000 UTC m=+613.186776412"
Sep 19 11:43:49.067058 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:49.067018    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:49.122596 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:49.122566    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba69ec7a-0643-4b63-9f60-093bd4a10f26-maas-api-tls\") pod \"maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f\" (UID: \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"
Sep 19 11:43:49.125274 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:49.125237    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba69ec7a-0643-4b63-9f60-093bd4a10f26-maas-api-tls\") pod \"maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f\" (UID: \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"
Sep 19 11:43:49.132877 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:49.132855    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6456-47f5f7"
Sep 19 11:43:49.138809 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:49.138782    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:49.223093 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:49.222995    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q9x7\" (UniqueName: \"kubernetes.io/projected/89457412-d902-4632-ba3e-4f6357e7338b-kube-api-access-8q9x7\") pod \"89457412-d902-4632-ba3e-4f6357e7338b\" (UID: \"89457412-d902-4632-ba3e-4f6357e7338b\") "
Sep 19 11:43:49.225432 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:49.225407    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89457412-d902-4632-ba3e-4f6357e7338b-kube-api-access-8q9x7" (OuterVolumeSpecName: "kube-api-access-8q9x7") pod "89457412-d902-4632-ba3e-4f6357e7338b" (UID: "89457412-d902-4632-ba3e-4f6357e7338b"). InnerVolumeSpecName "kube-api-access-8q9x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:43:49.324485 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:49.324446    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q9x7\" (UniqueName: \"kubernetes.io/projected/89457412-d902-4632-ba3e-4f6357e7338b-kube-api-access-8q9x7\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:43:49.352891 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:49.352850    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"
Sep 19 11:43:49.566708 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:49.566675    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:49.632730 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:49.632696    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"]
Sep 19 11:43:49.635357 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:43:49.635325    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba69ec7a_0643_4b63_9f60_093bd4a10f26.slice/crio-c1b8c09e1c1bd274591d324643e814fa8db184886d0297a48b2092a25f2eef40 WatchSource:0}: Error finding container c1b8c09e1c1bd274591d324643e814fa8db184886d0297a48b2092a25f2eef40: Status 404 returned error can't find the container with id c1b8c09e1c1bd274591d324643e814fa8db184886d0297a48b2092a25f2eef40
Sep 19 11:43:49.903833 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:49.903750    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89457412-d902-4632-ba3e-4f6357e7338b" path="/var/lib/kubelet/pods/89457412-d902-4632-ba3e-4f6357e7338b/volumes"
Sep 19 11:43:49.974583 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:49.974555    2584 generic.go:356] "Generic (PLEG): container finished" podID="89457412-d902-4632-ba3e-4f6357e7338b" containerID="1a6ea20cd8a541621a9d94d367d79e4f53d53f92ec0344a8397471d9b06b7654" exitCode=137
Sep 19 11:43:49.974723 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:49.974597    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6456-47f5f7"
Sep 19 11:43:49.974723 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:49.974653    2584 scope.go:117] "RemoveContainer" containerID="1a6ea20cd8a541621a9d94d367d79e4f53d53f92ec0344a8397471d9b06b7654"
Sep 19 11:43:49.976241 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:49.976216    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f" event={"ID":"ba69ec7a-0643-4b63-9f60-093bd4a10f26","Type":"ContainerStarted","Data":"c1b8c09e1c1bd274591d324643e814fa8db184886d0297a48b2092a25f2eef40"}
Sep 19 11:43:49.976454 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:49.976434    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"
Sep 19 11:43:49.991269 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:49.991239    2584 scope.go:117] "RemoveContainer" containerID="1a6ea20cd8a541621a9d94d367d79e4f53d53f92ec0344a8397471d9b06b7654"
Sep 19 11:43:49.991648 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:43:49.991620    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a6ea20cd8a541621a9d94d367d79e4f53d53f92ec0344a8397471d9b06b7654\": container with ID starting with 1a6ea20cd8a541621a9d94d367d79e4f53d53f92ec0344a8397471d9b06b7654 not found: ID does not exist" containerID="1a6ea20cd8a541621a9d94d367d79e4f53d53f92ec0344a8397471d9b06b7654"
Sep 19 11:43:49.991739 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:49.991660    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a6ea20cd8a541621a9d94d367d79e4f53d53f92ec0344a8397471d9b06b7654"} err="failed to get container status \"1a6ea20cd8a541621a9d94d367d79e4f53d53f92ec0344a8397471d9b06b7654\": rpc error: code = NotFound desc = could not find container \"1a6ea20cd8a541621a9d94d367d79e4f53d53f92ec0344a8397471d9b06b7654\": container with ID starting with 1a6ea20cd8a541621a9d94d367d79e4f53d53f92ec0344a8397471d9b06b7654 not found: ID does not exist"
Sep 19 11:43:50.082219 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:50.082187    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:50.166904 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:50.166820    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:50.457785 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:50.457696    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:50.984988 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:50.984955    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f" event={"ID":"ba69ec7a-0643-4b63-9f60-093bd4a10f26","Type":"ContainerStarted","Data":"0dc55cda4977d80155a7fd168eb80844b28ed08bd98c1a537c511e7df9ba7fa0"}
Sep 19 11:43:50.985348 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:50.985174    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"
Sep 19 11:43:51.003328 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:51.003279    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f" podStartSLOduration=2.675077764 podStartE2EDuration="3.003260541s" totalImagesPullingTime="328.182777ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:43:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:43:51.001907238 +0000 UTC m=+615.201937071" watchObservedRunningTime="2026-09-19 11:43:51.003260541 +0000 UTC m=+615.203290343"
Sep 19 11:43:51.131092 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:51.131064    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6456-d5d5af"
Sep 19 11:43:51.243126 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:51.243050    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hclk\" (UniqueName: \"kubernetes.io/projected/037662b6-e632-4c81-8894-f6f14a1ffe01-kube-api-access-4hclk\") pod \"037662b6-e632-4c81-8894-f6f14a1ffe01\" (UID: \"037662b6-e632-4c81-8894-f6f14a1ffe01\") "
Sep 19 11:43:51.245325 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:51.245292    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037662b6-e632-4c81-8894-f6f14a1ffe01-kube-api-access-4hclk" (OuterVolumeSpecName: "kube-api-access-4hclk") pod "037662b6-e632-4c81-8894-f6f14a1ffe01" (UID: "037662b6-e632-4c81-8894-f6f14a1ffe01"). InnerVolumeSpecName "kube-api-access-4hclk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:43:51.255348 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:51.255301    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:51.344278 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:51.344244    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hclk\" (UniqueName: \"kubernetes.io/projected/037662b6-e632-4c81-8894-f6f14a1ffe01-kube-api-access-4hclk\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:43:51.657859 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:51.657815    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:51.757063 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:51.757029    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:51.903875 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:51.903838    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="037662b6-e632-4c81-8894-f6f14a1ffe01" path="/var/lib/kubelet/pods/037662b6-e632-4c81-8894-f6f14a1ffe01/volumes"
Sep 19 11:43:51.992336 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:51.992303    2584 generic.go:356] "Generic (PLEG): container finished" podID="037662b6-e632-4c81-8894-f6f14a1ffe01" containerID="845237a22738816c7a8dd98adbc5a7c2faa25c7b1e80a9e32c9f9e69af13602e" exitCode=137
Sep 19 11:43:51.992823 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:51.992345    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6456-d5d5af"
Sep 19 11:43:51.992823 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:51.992350    2584 scope.go:117] "RemoveContainer" containerID="845237a22738816c7a8dd98adbc5a7c2faa25c7b1e80a9e32c9f9e69af13602e"
Sep 19 11:43:52.001382 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:52.001364    2584 scope.go:117] "RemoveContainer" containerID="845237a22738816c7a8dd98adbc5a7c2faa25c7b1e80a9e32c9f9e69af13602e"
Sep 19 11:43:52.001681 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:43:52.001662    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845237a22738816c7a8dd98adbc5a7c2faa25c7b1e80a9e32c9f9e69af13602e\": container with ID starting with 845237a22738816c7a8dd98adbc5a7c2faa25c7b1e80a9e32c9f9e69af13602e not found: ID does not exist" containerID="845237a22738816c7a8dd98adbc5a7c2faa25c7b1e80a9e32c9f9e69af13602e"
Sep 19 11:43:52.001743 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:52.001693    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845237a22738816c7a8dd98adbc5a7c2faa25c7b1e80a9e32c9f9e69af13602e"} err="failed to get container status \"845237a22738816c7a8dd98adbc5a7c2faa25c7b1e80a9e32c9f9e69af13602e\": rpc error: code = NotFound desc = could not find container \"845237a22738816c7a8dd98adbc5a7c2faa25c7b1e80a9e32c9f9e69af13602e\": container with ID starting with 845237a22738816c7a8dd98adbc5a7c2faa25c7b1e80a9e32c9f9e69af13602e not found: ID does not exist"
Sep 19 11:43:52.241387 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:52.241275    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:52.965193 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:52.965160    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"
Sep 19 11:43:53.145064 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.145027    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"]
Sep 19 11:43:53.150116 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.150097    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"
Sep 19 11:43:53.152503 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.152481    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-7a524f-serving-cert\""
Sep 19 11:43:53.156507 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.156481    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"]
Sep 19 11:43:53.229038 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.228940    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:53.261513 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.261468    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5aa702-65ad-4c79-ae23-57542b7a195e-openshift-service-ca\") pod \"maas-api-e2e-autores-7a524f-8479d97798-bc798\" (UID: \"1f5aa702-65ad-4c79-ae23-57542b7a195e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"
Sep 19 11:43:53.261698 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.261576    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtcmz\" (UniqueName: \"kubernetes.io/projected/1f5aa702-65ad-4c79-ae23-57542b7a195e-kube-api-access-vtcmz\") pod \"maas-api-e2e-autores-7a524f-8479d97798-bc798\" (UID: \"1f5aa702-65ad-4c79-ae23-57542b7a195e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"
Sep 19 11:43:53.261698 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.261595    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f5aa702-65ad-4c79-ae23-57542b7a195e-maas-api-tls\") pod \"maas-api-e2e-autores-7a524f-8479d97798-bc798\" (UID: \"1f5aa702-65ad-4c79-ae23-57542b7a195e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"
Sep 19 11:43:53.261698 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.261610    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f5aa702-65ad-4c79-ae23-57542b7a195e-metrics-certs\") pod \"maas-api-e2e-autores-7a524f-8479d97798-bc798\" (UID: \"1f5aa702-65ad-4c79-ae23-57542b7a195e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"
Sep 19 11:43:53.329440 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.329383    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:53.362200 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.362158    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtcmz\" (UniqueName: \"kubernetes.io/projected/1f5aa702-65ad-4c79-ae23-57542b7a195e-kube-api-access-vtcmz\") pod \"maas-api-e2e-autores-7a524f-8479d97798-bc798\" (UID: \"1f5aa702-65ad-4c79-ae23-57542b7a195e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"
Sep 19 11:43:53.362200 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.362204    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f5aa702-65ad-4c79-ae23-57542b7a195e-maas-api-tls\") pod \"maas-api-e2e-autores-7a524f-8479d97798-bc798\" (UID: \"1f5aa702-65ad-4c79-ae23-57542b7a195e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"
Sep 19 11:43:53.362475 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.362233    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f5aa702-65ad-4c79-ae23-57542b7a195e-metrics-certs\") pod \"maas-api-e2e-autores-7a524f-8479d97798-bc798\" (UID: \"1f5aa702-65ad-4c79-ae23-57542b7a195e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"
Sep 19 11:43:53.362475 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.362304    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5aa702-65ad-4c79-ae23-57542b7a195e-openshift-service-ca\") pod \"maas-api-e2e-autores-7a524f-8479d97798-bc798\" (UID: \"1f5aa702-65ad-4c79-ae23-57542b7a195e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"
Sep 19 11:43:53.362475 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:43:53.362326    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-serving-cert: secret "maas-api-e2e-autores-7a524f-serving-cert" not found
Sep 19 11:43:53.362475 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:43:53.362397    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f5aa702-65ad-4c79-ae23-57542b7a195e-maas-api-tls podName:1f5aa702-65ad-4c79-ae23-57542b7a195e nodeName:}" failed. No retries permitted until 2026-09-19 11:43:53.862377003 +0000 UTC m=+618.062406789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f5aa702-65ad-4c79-ae23-57542b7a195e-maas-api-tls") pod "maas-api-e2e-autores-7a524f-8479d97798-bc798" (UID: "1f5aa702-65ad-4c79-ae23-57542b7a195e") : secret "maas-api-e2e-autores-7a524f-serving-cert" not found
Sep 19 11:43:53.362997 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.362976    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5aa702-65ad-4c79-ae23-57542b7a195e-openshift-service-ca\") pod \"maas-api-e2e-autores-7a524f-8479d97798-bc798\" (UID: \"1f5aa702-65ad-4c79-ae23-57542b7a195e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"
Sep 19 11:43:53.365022 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.365001    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f5aa702-65ad-4c79-ae23-57542b7a195e-metrics-certs\") pod \"maas-api-e2e-autores-7a524f-8479d97798-bc798\" (UID: \"1f5aa702-65ad-4c79-ae23-57542b7a195e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"
Sep 19 11:43:53.370698 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.370670    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtcmz\" (UniqueName: \"kubernetes.io/projected/1f5aa702-65ad-4c79-ae23-57542b7a195e-kube-api-access-vtcmz\") pod \"maas-api-e2e-autores-7a524f-8479d97798-bc798\" (UID: \"1f5aa702-65ad-4c79-ae23-57542b7a195e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"
Sep 19 11:43:53.488906 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.488786    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:53.686302 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.686272    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:53.791612 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.791509    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:53.868086 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.868047    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f5aa702-65ad-4c79-ae23-57542b7a195e-maas-api-tls\") pod \"maas-api-e2e-autores-7a524f-8479d97798-bc798\" (UID: \"1f5aa702-65ad-4c79-ae23-57542b7a195e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"
Sep 19 11:43:53.870964 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:53.870936    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f5aa702-65ad-4c79-ae23-57542b7a195e-maas-api-tls\") pod \"maas-api-e2e-autores-7a524f-8479d97798-bc798\" (UID: \"1f5aa702-65ad-4c79-ae23-57542b7a195e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"
Sep 19 11:43:54.063969 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:54.063939    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"
Sep 19 11:43:54.173270 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:54.173233    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:54.345789 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:54.345754    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"]
Sep 19 11:43:54.346877 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:43:54.346849    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5aa702_65ad_4c79_ae23_57542b7a195e.slice/crio-00ca1d0c93d5c9c21dbdf2bf2fe49888bb39317f04fe137fa69670038d7edef3 WatchSource:0}: Error finding container 00ca1d0c93d5c9c21dbdf2bf2fe49888bb39317f04fe137fa69670038d7edef3: Status 404 returned error can't find the container with id 00ca1d0c93d5c9c21dbdf2bf2fe49888bb39317f04fe137fa69670038d7edef3
Sep 19 11:43:54.976143 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:54.976034    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:55.015171 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.015129    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798" event={"ID":"1f5aa702-65ad-4c79-ae23-57542b7a195e","Type":"ContainerStarted","Data":"22b02028fb7d24984449a6f54691794f01c2c989328fafe0860c3fc9c6c010a2"}
Sep 19 11:43:55.015360 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.015181    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798" event={"ID":"1f5aa702-65ad-4c79-ae23-57542b7a195e","Type":"ContainerStarted","Data":"00ca1d0c93d5c9c21dbdf2bf2fe49888bb39317f04fe137fa69670038d7edef3"}
Sep 19 11:43:55.015360 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.015203    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"
Sep 19 11:43:55.034190 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.034133    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798" podStartSLOduration=1.6748385030000001 podStartE2EDuration="2.034114997s" totalImagesPullingTime="359.276494ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:43:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:43:55.031270389 +0000 UTC m=+619.231300194" watchObservedRunningTime="2026-09-19 11:43:55.034114997 +0000 UTC m=+619.234144800"
Sep 19 11:43:55.035535 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.035500    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:55.091185 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.091153    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:55.558446 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.558399    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:55.660117 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.660082    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:55.745961 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.745925    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46"]
Sep 19 11:43:55.749566 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.749543    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46"
Sep 19 11:43:55.752138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.752113    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-cb615d6f-dockercfg-8x7fx\""
Sep 19 11:43:55.752138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.752124    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-cb615d6f\""
Sep 19 11:43:55.758473 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.758445    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46"]
Sep 19 11:43:55.887779 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.887730    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqxxl\" (UniqueName: \"kubernetes.io/projected/69ad4210-5ed6-433c-922d-6341e9c7e4d7-kube-api-access-sqxxl\") pod \"payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46\" (UID: \"69ad4210-5ed6-433c-922d-6341e9c7e4d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46"
Sep 19 11:43:55.888306 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.888282    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69ad4210-5ed6-433c-922d-6341e9c7e4d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46\" (UID: \"69ad4210-5ed6-433c-922d-6341e9c7e4d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46"
Sep 19 11:43:55.989361 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.989308    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69ad4210-5ed6-433c-922d-6341e9c7e4d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46\" (UID: \"69ad4210-5ed6-433c-922d-6341e9c7e4d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46"
Sep 19 11:43:55.989577 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.989448    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqxxl\" (UniqueName: \"kubernetes.io/projected/69ad4210-5ed6-433c-922d-6341e9c7e4d7-kube-api-access-sqxxl\") pod \"payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46\" (UID: \"69ad4210-5ed6-433c-922d-6341e9c7e4d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46"
Sep 19 11:43:55.990772 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.990734    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69ad4210-5ed6-433c-922d-6341e9c7e4d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46\" (UID: \"69ad4210-5ed6-433c-922d-6341e9c7e4d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46"
Sep 19 11:43:55.993171 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.993144    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"
Sep 19 11:43:55.999897 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:55.999851    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqxxl\" (UniqueName: \"kubernetes.io/projected/69ad4210-5ed6-433c-922d-6341e9c7e4d7-kube-api-access-sqxxl\") pod \"payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46\" (UID: \"69ad4210-5ed6-433c-922d-6341e9c7e4d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46"
Sep 19 11:43:56.060725 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:56.060694    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46"
Sep 19 11:43:56.272199 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:56.272110    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:56.354490 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:56.354465    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46"]
Sep 19 11:43:56.357001 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:43:56.356973    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ad4210_5ed6_433c_922d_6341e9c7e4d7.slice/crio-8b839f55e2fcf738af07b133fb5a5fb451ebc31c27570d74c9dc8d83d3a54c5b WatchSource:0}: Error finding container 8b839f55e2fcf738af07b133fb5a5fb451ebc31c27570d74c9dc8d83d3a54c5b: Status 404 returned error can't find the container with id 8b839f55e2fcf738af07b133fb5a5fb451ebc31c27570d74c9dc8d83d3a54c5b
Sep 19 11:43:57.018128 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:57.018092    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"
Sep 19 11:43:57.030146 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:57.030107    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46" event={"ID":"69ad4210-5ed6-433c-922d-6341e9c7e4d7","Type":"ContainerStarted","Data":"8b839f55e2fcf738af07b133fb5a5fb451ebc31c27570d74c9dc8d83d3a54c5b"}
Sep 19 11:43:57.082588 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:57.082554    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:58.039364 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:58.039262    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46" event={"ID":"69ad4210-5ed6-433c-922d-6341e9c7e4d7","Type":"ContainerStarted","Data":"525a01ed94bd3d03adb3091d4e071386b8fb494414c116edd8754c21edfa895e"}
Sep 19 11:43:58.039793 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:58.039402    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46"
Sep 19 11:43:58.058053 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:58.058006    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46" podStartSLOduration=1.7653460810000001 podStartE2EDuration="3.057990971s" totalImagesPullingTime="1.29264489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:43:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:43:58.054860155 +0000 UTC m=+622.254889958" watchObservedRunningTime="2026-09-19 11:43:58.057990971 +0000 UTC m=+622.258020772"
Sep 19 11:43:58.067044 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:58.067020    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:58.847877 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:58.847839    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:43:59.691755 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:43:59.691711    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:00.471604 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:00.471572    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:01.029418 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:01.029392    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"
Sep 19 11:44:01.530496 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:01.530467    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:02.285988 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:02.285951    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:02.383804 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:02.383770    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:02.649652 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:02.649615    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:02.974205 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:02.974114    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:03.247085 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:03.246999    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:03.347501 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:03.347463    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:03.912247 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:03.912210    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:04.013569 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:04.013529    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:04.142169 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:04.142139    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:04.242461 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:04.242438    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:04.342625 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:04.342594    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:04.659022 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:04.658986    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:04.761488 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:04.761454    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:05.085134 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:05.085103    2584 generic.go:356] "Generic (PLEG): container finished" podID="4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" containerID="9578c3594ebf5961bb2ff5b975165155e314f25f3817200aefa5bcf5a2392ab7" exitCode=0
Sep 19 11:44:05.085398 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:05.085188    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng" event={"ID":"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088","Type":"ContainerDied","Data":"9578c3594ebf5961bb2ff5b975165155e314f25f3817200aefa5bcf5a2392ab7"}
Sep 19 11:44:05.085793 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:05.085725    2584 scope.go:117] "RemoveContainer" containerID="9578c3594ebf5961bb2ff5b975165155e314f25f3817200aefa5bcf5a2392ab7"
Sep 19 11:44:05.087966 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:05.087930    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:44:05.088061 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:05.088006    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:44:05.088061 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:05.088038    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:44:05.129066 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:05.129033    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:05.231121 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:05.231065    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:05.292778 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:05.292729    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:05.448455 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:05.448379    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:05.629603 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:05.629570    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:05.791545 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:05.791439    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:05.892736 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:05.892697    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:06.048859 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:06.048731    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:06.095574 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:06.095534    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng" event={"ID":"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088","Type":"ContainerStarted","Data":"69bdf708f919872c437c67f41392bc4b3e5a794c846bfc5c1936f62a182e9347"}
Sep 19 11:44:06.161449 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:06.161420    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:06.308054 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:06.307954    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:06.466121 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:06.466076    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:44:06.466326 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:06.466306    2584 patch_prober.go:28] interesting pod/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 19 11:44:06.466373 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:06.466357    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng" podUID="4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 19 11:44:06.776471 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:06.776428    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:07.466470 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:07.466427    2584 patch_prober.go:28] interesting pod/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 19 11:44:07.466945 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:07.466525    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng" podUID="4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 19 11:44:07.876603 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:07.876571    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:08.466624 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:08.466582    2584 patch_prober.go:28] interesting pod/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 19 11:44:08.467067 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:08.466655    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng" podUID="4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 19 11:44:08.598267 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:08.598228    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:08.699438 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:08.699374    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:09.048749 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:09.048712    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46"
Sep 19 11:44:09.460074 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:09.460040    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:09.466029 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:09.465996    2584 patch_prober.go:28] interesting pod/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 19 11:44:09.466176 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:09.466055    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng" podUID="4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 19 11:44:10.061781 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.061747    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:10.169416 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.169376    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:10.258865 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.258826    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:10.259542 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.259518    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"]
Sep 19 11:44:10.264942 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.264890    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"
Sep 19 11:44:10.268175 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.268152    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d8d8bb-serving-cert\""
Sep 19 11:44:10.275706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.275680    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"]
Sep 19 11:44:10.327823 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.327738    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:10.356823 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.356789    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-maas-api-tls\") pod \"maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx\" (UID: \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"
Sep 19 11:44:10.357021 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.356860    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-metrics-certs\") pod \"maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx\" (UID: \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"
Sep 19 11:44:10.357021 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.356883    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx\" (UID: \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"
Sep 19 11:44:10.357021 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.356903    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhsnd\" (UniqueName: \"kubernetes.io/projected/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-kube-api-access-hhsnd\") pod \"maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx\" (UID: \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"
Sep 19 11:44:10.427140 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.427103    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:10.458547 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.458507    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-metrics-certs\") pod \"maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx\" (UID: \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"
Sep 19 11:44:10.458740 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.458570    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx\" (UID: \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"
Sep 19 11:44:10.458740 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.458606    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhsnd\" (UniqueName: \"kubernetes.io/projected/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-kube-api-access-hhsnd\") pod \"maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx\" (UID: \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"
Sep 19 11:44:10.458740 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.458700    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-maas-api-tls\") pod \"maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx\" (UID: \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"
Sep 19 11:44:10.459370 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.459350    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx\" (UID: \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"
Sep 19 11:44:10.461189 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.461165    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-metrics-certs\") pod \"maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx\" (UID: \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"
Sep 19 11:44:10.461305 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.461284    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-maas-api-tls\") pod \"maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx\" (UID: \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"
Sep 19 11:44:10.466327 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.466301    2584 patch_prober.go:28] interesting pod/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 19 11:44:10.466409 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.466362    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng" podUID="4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 19 11:44:10.470043 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.470023    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhsnd\" (UniqueName: \"kubernetes.io/projected/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-kube-api-access-hhsnd\") pod \"maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx\" (UID: \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"
Sep 19 11:44:10.526645 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.526615    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:10.577973 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.577872    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"
Sep 19 11:44:10.879752 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:10.879729    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"]
Sep 19 11:44:10.881619 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:44:10.881588    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe16a33a_bb0c_4330_a9b4_0c2d8c53b04f.slice/crio-fe98962c496d7765b8df57efef4a52305aeb8d54b05609f36ced791166c09d7d WatchSource:0}: Error finding container fe98962c496d7765b8df57efef4a52305aeb8d54b05609f36ced791166c09d7d: Status 404 returned error can't find the container with id fe98962c496d7765b8df57efef4a52305aeb8d54b05609f36ced791166c09d7d
Sep 19 11:44:11.066146 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:11.066112    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:11.136969 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:11.136860    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx" event={"ID":"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f","Type":"ContainerStarted","Data":"fe98962c496d7765b8df57efef4a52305aeb8d54b05609f36ced791166c09d7d"}
Sep 19 11:44:11.164930 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:11.164878    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:11.471606 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:11.471334    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:44:11.471800 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:11.471773    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:44:11.472661 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:11.472626    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:44:11.477165 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:11.477136    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:12.144130 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:12.144084    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx" event={"ID":"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f","Type":"ContainerStarted","Data":"403f9aa250e46a81669c6327cacce24c5d43fc8a2ae0806613e2caa30a080fbe"}
Sep 19 11:44:12.165959 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:12.165890    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx" podStartSLOduration=1.686508345 podStartE2EDuration="2.165872582s" totalImagesPullingTime="479.364237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:44:12.162829803 +0000 UTC m=+636.362859605" watchObservedRunningTime="2026-09-19 11:44:12.165872582 +0000 UTC m=+636.365902384"
Sep 19 11:44:12.280090 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:12.280057    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:13.149280 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:13.149244    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"
Sep 19 11:44:13.166742 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:13.166703    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:13.628516 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:13.628480    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:13.885523 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:13.885441    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:14.744144 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:14.744108    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:15.635097 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:15.635065    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:16.398628 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:16.398592    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:16.501644 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:16.501599    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:16.599567 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:16.599527    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:17.489855 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:17.489814    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:19.161002 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:19.160970    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"
Sep 19 11:44:20.809951 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:20.809905    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:20.852975 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:20.852906    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46"]
Sep 19 11:44:20.853187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:20.853160    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46" podUID="69ad4210-5ed6-433c-922d-6341e9c7e4d7" containerName="payload-pre-processing" containerID="cri-o://525a01ed94bd3d03adb3091d4e071386b8fb494414c116edd8754c21edfa895e" gracePeriod=30
Sep 19 11:44:20.908196 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:20.908162    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:21.008104 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.008073    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:21.065135 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.065038    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:21.120844 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.120809    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46"
Sep 19 11:44:21.180377 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.180346    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:21.211272 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.211238    2584 generic.go:356] "Generic (PLEG): container finished" podID="69ad4210-5ed6-433c-922d-6341e9c7e4d7" containerID="525a01ed94bd3d03adb3091d4e071386b8fb494414c116edd8754c21edfa895e" exitCode=0
Sep 19 11:44:21.211461 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.211292    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46"
Sep 19 11:44:21.211461 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.211323    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46" event={"ID":"69ad4210-5ed6-433c-922d-6341e9c7e4d7","Type":"ContainerDied","Data":"525a01ed94bd3d03adb3091d4e071386b8fb494414c116edd8754c21edfa895e"}
Sep 19 11:44:21.211461 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.211380    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46" event={"ID":"69ad4210-5ed6-433c-922d-6341e9c7e4d7","Type":"ContainerDied","Data":"8b839f55e2fcf738af07b133fb5a5fb451ebc31c27570d74c9dc8d83d3a54c5b"}
Sep 19 11:44:21.211461 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.211407    2584 scope.go:117] "RemoveContainer" containerID="525a01ed94bd3d03adb3091d4e071386b8fb494414c116edd8754c21edfa895e"
Sep 19 11:44:21.221185 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.221164    2584 scope.go:117] "RemoveContainer" containerID="525a01ed94bd3d03adb3091d4e071386b8fb494414c116edd8754c21edfa895e"
Sep 19 11:44:21.221499 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:44:21.221477    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"525a01ed94bd3d03adb3091d4e071386b8fb494414c116edd8754c21edfa895e\": container with ID starting with 525a01ed94bd3d03adb3091d4e071386b8fb494414c116edd8754c21edfa895e not found: ID does not exist" containerID="525a01ed94bd3d03adb3091d4e071386b8fb494414c116edd8754c21edfa895e"
Sep 19 11:44:21.221580 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.221511    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525a01ed94bd3d03adb3091d4e071386b8fb494414c116edd8754c21edfa895e"} err="failed to get container status \"525a01ed94bd3d03adb3091d4e071386b8fb494414c116edd8754c21edfa895e\": rpc error: code = NotFound desc = could not find container \"525a01ed94bd3d03adb3091d4e071386b8fb494414c116edd8754c21edfa895e\": container with ID starting with 525a01ed94bd3d03adb3091d4e071386b8fb494414c116edd8754c21edfa895e not found: ID does not exist"
Sep 19 11:44:21.261222 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.261197    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69ad4210-5ed6-433c-922d-6341e9c7e4d7-plugins-config-volume\") pod \"69ad4210-5ed6-433c-922d-6341e9c7e4d7\" (UID: \"69ad4210-5ed6-433c-922d-6341e9c7e4d7\") "
Sep 19 11:44:21.261381 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.261241    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqxxl\" (UniqueName: \"kubernetes.io/projected/69ad4210-5ed6-433c-922d-6341e9c7e4d7-kube-api-access-sqxxl\") pod \"69ad4210-5ed6-433c-922d-6341e9c7e4d7\" (UID: \"69ad4210-5ed6-433c-922d-6341e9c7e4d7\") "
Sep 19 11:44:21.261695 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.261651    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ad4210-5ed6-433c-922d-6341e9c7e4d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69ad4210-5ed6-433c-922d-6341e9c7e4d7" (UID: "69ad4210-5ed6-433c-922d-6341e9c7e4d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:44:21.263729 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.263704    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ad4210-5ed6-433c-922d-6341e9c7e4d7-kube-api-access-sqxxl" (OuterVolumeSpecName: "kube-api-access-sqxxl") pod "69ad4210-5ed6-433c-922d-6341e9c7e4d7" (UID: "69ad4210-5ed6-433c-922d-6341e9c7e4d7"). InnerVolumeSpecName "kube-api-access-sqxxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:44:21.362493 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.362377    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqxxl\" (UniqueName: \"kubernetes.io/projected/69ad4210-5ed6-433c-922d-6341e9c7e4d7-kube-api-access-sqxxl\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:44:21.362493 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.362412    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69ad4210-5ed6-433c-922d-6341e9c7e4d7-plugins-config-volume\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:44:21.532648 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.532613    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46"]
Sep 19 11:44:21.535698 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.535667    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cb615d6f-77879547d-hwq46"]
Sep 19 11:44:21.575084 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.575043    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:21.678522 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.678433    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:21.776711 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.776674    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:21.859758 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.859725    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:21.904864 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:21.904827    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69ad4210-5ed6-433c-922d-6341e9c7e4d7" path="/var/lib/kubelet/pods/69ad4210-5ed6-433c-922d-6341e9c7e4d7/volumes"
Sep 19 11:44:22.185976 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:22.185900    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:22.519340 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:22.519253    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:22.619644 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:22.619609    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:23.429788 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:23.429755    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:24.345526 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:24.345492    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:24.444414 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:24.444383    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:24.498924 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:24.498877    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:25.444358 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:25.444326    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:26.346885 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:26.346853    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:27.368171 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.368137    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"]
Sep 19 11:44:27.368592 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.368545    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69ad4210-5ed6-433c-922d-6341e9c7e4d7" containerName="payload-pre-processing"
Sep 19 11:44:27.368592 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.368556    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="69ad4210-5ed6-433c-922d-6341e9c7e4d7" containerName="payload-pre-processing"
Sep 19 11:44:27.368679 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.368648    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="69ad4210-5ed6-433c-922d-6341e9c7e4d7" containerName="payload-pre-processing"
Sep 19 11:44:27.373077 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.373052    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"
Sep 19 11:44:27.375696 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.375672    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-9cf8a188-serving-cert\""
Sep 19 11:44:27.391147 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.391108    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"]
Sep 19 11:44:27.459311 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.459280    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:27.521872 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.521831    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-maas-api-tls\") pod \"maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh\" (UID: \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"
Sep 19 11:44:27.521872 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.521873    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g44gc\" (UniqueName: \"kubernetes.io/projected/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-kube-api-access-g44gc\") pod \"maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh\" (UID: \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"
Sep 19 11:44:27.522119 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.521898    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-openshift-service-ca\") pod \"maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh\" (UID: \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"
Sep 19 11:44:27.522119 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.521995    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-metrics-certs\") pod \"maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh\" (UID: \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"
Sep 19 11:44:27.561562 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.561531    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:27.623212 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.623116    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-maas-api-tls\") pod \"maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh\" (UID: \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"
Sep 19 11:44:27.623212 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.623151    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g44gc\" (UniqueName: \"kubernetes.io/projected/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-kube-api-access-g44gc\") pod \"maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh\" (UID: \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"
Sep 19 11:44:27.623212 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.623175    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-openshift-service-ca\") pod \"maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh\" (UID: \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"
Sep 19 11:44:27.623474 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.623231    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-metrics-certs\") pod \"maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh\" (UID: \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"
Sep 19 11:44:27.623474 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:44:27.623299    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-serving-cert: secret "maas-api-e2e-migrate-9cf8a188-serving-cert" not found
Sep 19 11:44:27.623474 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:44:27.623365    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-maas-api-tls podName:459f1af3-3fba-4f1d-8edb-890d4e98d8ca nodeName:}" failed. No retries permitted until 2026-09-19 11:44:28.123344243 +0000 UTC m=+652.323374023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-maas-api-tls") pod "maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh" (UID: "459f1af3-3fba-4f1d-8edb-890d4e98d8ca") : secret "maas-api-e2e-migrate-9cf8a188-serving-cert" not found
Sep 19 11:44:27.623838 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.623818    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-openshift-service-ca\") pod \"maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh\" (UID: \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"
Sep 19 11:44:27.626034 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.626010    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-metrics-certs\") pod \"maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh\" (UID: \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"
Sep 19 11:44:27.631619 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.631593    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g44gc\" (UniqueName: \"kubernetes.io/projected/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-kube-api-access-g44gc\") pod \"maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh\" (UID: \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"
Sep 19 11:44:27.697624 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:27.697594    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:28.027314 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:28.027218    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:28.128115 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:28.128074    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-maas-api-tls\") pod \"maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh\" (UID: \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"
Sep 19 11:44:28.128835 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:28.128800    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:28.130983 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:28.130959    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-maas-api-tls\") pod \"maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh\" (UID: \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"
Sep 19 11:44:28.199025 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:28.198992    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:28.284785 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:28.284686    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"
Sep 19 11:44:28.568228 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:28.568199    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"]
Sep 19 11:44:28.570119 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:44:28.570084    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459f1af3_3fba_4f1d_8edb_890d4e98d8ca.slice/crio-029c3d9ea298dee5c0a5a4f6edea99b795569730121a951894146d52d4616c2e WatchSource:0}: Error finding container 029c3d9ea298dee5c0a5a4f6edea99b795569730121a951894146d52d4616c2e: Status 404 returned error can't find the container with id 029c3d9ea298dee5c0a5a4f6edea99b795569730121a951894146d52d4616c2e
Sep 19 11:44:28.612690 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:28.612654    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:29.259050 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:29.258944    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh" event={"ID":"459f1af3-3fba-4f1d-8edb-890d4e98d8ca","Type":"ContainerStarted","Data":"7d52f61185fa6c88e145955206417b86474c7ad118b10abf7f4cab218d545a02"}
Sep 19 11:44:29.259050 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:29.259005    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh" event={"ID":"459f1af3-3fba-4f1d-8edb-890d4e98d8ca","Type":"ContainerStarted","Data":"029c3d9ea298dee5c0a5a4f6edea99b795569730121a951894146d52d4616c2e"}
Sep 19 11:44:29.259050 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:29.259030    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"
Sep 19 11:44:29.275615 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:29.275544    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh" podStartSLOduration=1.9022490539999999 podStartE2EDuration="2.275525486s" totalImagesPullingTime="373.276432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:44:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:44:29.27443503 +0000 UTC m=+653.474464845" watchObservedRunningTime="2026-09-19 11:44:29.275525486 +0000 UTC m=+653.475555285"
Sep 19 11:44:29.814693 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:29.814659    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:30.570261 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:30.570230    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:30.751475 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:30.751442    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:31.259697 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:31.259653    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:32.091577 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:32.091540    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:32.796479 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:32.796443    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:33.457414 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:33.457375    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:34.450335 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:34.450303    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:35.270290 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:35.270259    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"
Sep 19 11:44:48.501087 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:48.501050    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:49.976563 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:49.976528    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:50.971493 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:50.971456    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:51.717530 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:51.717496    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:52.382827 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:52.382790    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:53.160607 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:53.160575    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:53.792619 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:53.792587    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:53.892864 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:53.892825    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:54.451410 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:54.451374    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:55.340778 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:55.340736    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:55.969687 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:55.969627    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:56.732834 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:56.732800    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:57.535475 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:57.535440    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:58.201538 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:58.201494    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:58.978926 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:58.978889    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:44:59.764671 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:44:59.764638    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:05.907265 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:05.907232    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"]
Sep 19 11:45:05.908060 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:05.908022    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh" podUID="459f1af3-3fba-4f1d-8edb-890d4e98d8ca" containerName="maas-api" containerID="cri-o://7d52f61185fa6c88e145955206417b86474c7ad118b10abf7f4cab218d545a02" gracePeriod=30
Sep 19 11:45:05.938677 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:05.938630    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:06.032752 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.032726    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:06.133788 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.133750    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:06.169943 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.169839    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"
Sep 19 11:45:06.191837 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.191805    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:06.298110 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.298077    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:06.304166 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.304138    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g44gc\" (UniqueName: \"kubernetes.io/projected/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-kube-api-access-g44gc\") pod \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\" (UID: \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\") "
Sep 19 11:45:06.304324 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.304186    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-metrics-certs\") pod \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\" (UID: \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\") "
Sep 19 11:45:06.304324 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.304253    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-openshift-service-ca\") pod \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\" (UID: \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\") "
Sep 19 11:45:06.304324 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.304295    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-maas-api-tls\") pod \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\" (UID: \"459f1af3-3fba-4f1d-8edb-890d4e98d8ca\") "
Sep 19 11:45:06.304634 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.304611    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "459f1af3-3fba-4f1d-8edb-890d4e98d8ca" (UID: "459f1af3-3fba-4f1d-8edb-890d4e98d8ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:45:06.306637 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.306610    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-kube-api-access-g44gc" (OuterVolumeSpecName: "kube-api-access-g44gc") pod "459f1af3-3fba-4f1d-8edb-890d4e98d8ca" (UID: "459f1af3-3fba-4f1d-8edb-890d4e98d8ca"). InnerVolumeSpecName "kube-api-access-g44gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:45:06.306732 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.306612    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "459f1af3-3fba-4f1d-8edb-890d4e98d8ca" (UID: "459f1af3-3fba-4f1d-8edb-890d4e98d8ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:45:06.306732 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.306687    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "459f1af3-3fba-4f1d-8edb-890d4e98d8ca" (UID: "459f1af3-3fba-4f1d-8edb-890d4e98d8ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:45:06.405217 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.405184    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g44gc\" (UniqueName: \"kubernetes.io/projected/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-kube-api-access-g44gc\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:45:06.405217 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.405210    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-metrics-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:45:06.405217 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.405220    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-openshift-service-ca\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:45:06.405217 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.405228    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/459f1af3-3fba-4f1d-8edb-890d4e98d8ca-maas-api-tls\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:45:06.460894 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.460798    2584 generic.go:356] "Generic (PLEG): container finished" podID="459f1af3-3fba-4f1d-8edb-890d4e98d8ca" containerID="7d52f61185fa6c88e145955206417b86474c7ad118b10abf7f4cab218d545a02" exitCode=0
Sep 19 11:45:06.460894 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.460850    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"
Sep 19 11:45:06.460894 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.460871    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh" event={"ID":"459f1af3-3fba-4f1d-8edb-890d4e98d8ca","Type":"ContainerDied","Data":"7d52f61185fa6c88e145955206417b86474c7ad118b10abf7f4cab218d545a02"}
Sep 19 11:45:06.461117 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.460947    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh" event={"ID":"459f1af3-3fba-4f1d-8edb-890d4e98d8ca","Type":"ContainerDied","Data":"029c3d9ea298dee5c0a5a4f6edea99b795569730121a951894146d52d4616c2e"}
Sep 19 11:45:06.461117 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.460980    2584 scope.go:117] "RemoveContainer" containerID="7d52f61185fa6c88e145955206417b86474c7ad118b10abf7f4cab218d545a02"
Sep 19 11:45:06.473692 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.473651    2584 scope.go:117] "RemoveContainer" containerID="7d52f61185fa6c88e145955206417b86474c7ad118b10abf7f4cab218d545a02"
Sep 19 11:45:06.474025 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:45:06.474005    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d52f61185fa6c88e145955206417b86474c7ad118b10abf7f4cab218d545a02\": container with ID starting with 7d52f61185fa6c88e145955206417b86474c7ad118b10abf7f4cab218d545a02 not found: ID does not exist" containerID="7d52f61185fa6c88e145955206417b86474c7ad118b10abf7f4cab218d545a02"
Sep 19 11:45:06.474109 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.474041    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d52f61185fa6c88e145955206417b86474c7ad118b10abf7f4cab218d545a02"} err="failed to get container status \"7d52f61185fa6c88e145955206417b86474c7ad118b10abf7f4cab218d545a02\": rpc error: code = NotFound desc = could not find container \"7d52f61185fa6c88e145955206417b86474c7ad118b10abf7f4cab218d545a02\": container with ID starting with 7d52f61185fa6c88e145955206417b86474c7ad118b10abf7f4cab218d545a02 not found: ID does not exist"
Sep 19 11:45:06.484549 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.484521    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"]
Sep 19 11:45:06.488964 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.488936    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9cf8a188-7f87c669c7-9ptdh"]
Sep 19 11:45:06.588852 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.588794    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:06.739079 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:06.738983    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:07.017899 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.017809    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"]
Sep 19 11:45:07.018293 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.018277    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="459f1af3-3fba-4f1d-8edb-890d4e98d8ca" containerName="maas-api"
Sep 19 11:45:07.018391 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.018295    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="459f1af3-3fba-4f1d-8edb-890d4e98d8ca" containerName="maas-api"
Sep 19 11:45:07.018436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.018396    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="459f1af3-3fba-4f1d-8edb-890d4e98d8ca" containerName="maas-api"
Sep 19 11:45:07.023246 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.023217    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.025899 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.025875    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5deb747f-openshift-default-dockercfg-zhjmk\""
Sep 19 11:45:07.035523 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.035492    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"]
Sep 19 11:45:07.043257 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.043223    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:07.111895 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.111854    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-data\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.112114 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.111897    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-podinfo\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.112114 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.111999    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istiod-ca-cert\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.112114 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.112089    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-workload-certs\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.112291 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.112120    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv5mb\" (UniqueName: \"kubernetes.io/projected/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-kube-api-access-hv5mb\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.112291 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.112169    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-credential-socket\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.112291 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.112194    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-token\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.112291 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.112236    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-workload-socket\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.112291 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.112272    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-envoy\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.198493 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.198462    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:07.213856 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.213826    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-data\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.213993 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.213864    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-podinfo\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.213993 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.213894    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istiod-ca-cert\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.214102 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.213997    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-workload-certs\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.214102 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.214051    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv5mb\" (UniqueName: \"kubernetes.io/projected/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-kube-api-access-hv5mb\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.214102 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.214096    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-credential-socket\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.214251 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.214121    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-token\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.214251 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.214149    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-workload-socket\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.214251 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.214188    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-envoy\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.221874 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.214489    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-data\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.221874 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.214852    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-credential-socket\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.221874 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.214982    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-workload-socket\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.221874 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.215094    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-workload-certs\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.221874 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.215132    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istiod-ca-cert\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.221874 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.217426    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-envoy\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.223789 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.223749    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-podinfo\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.224426 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.224402    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-token\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.225776 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.225745    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv5mb\" (UniqueName: \"kubernetes.io/projected/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-kube-api-access-hv5mb\") pod \"e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") " pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.335697 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.335653    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:07.448403 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.448368    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:07.658040 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.658015    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"]
Sep 19 11:45:07.660057 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:45:07.660029    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39aa8a3_2686_4cc1_a684_82b2ad9a5f8f.slice/crio-aa6471114eb4547eb8fe87c0fce0b5801b3d938fec828d50aa8a97f9e8f5a011 WatchSource:0}: Error finding container aa6471114eb4547eb8fe87c0fce0b5801b3d938fec828d50aa8a97f9e8f5a011: Status 404 returned error can't find the container with id aa6471114eb4547eb8fe87c0fce0b5801b3d938fec828d50aa8a97f9e8f5a011
Sep 19 11:45:07.662539 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.662495    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:45:07.662622 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.662593    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:45:07.662672 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.662639    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:45:07.905218 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:07.905180    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="459f1af3-3fba-4f1d-8edb-890d4e98d8ca" path="/var/lib/kubelet/pods/459f1af3-3fba-4f1d-8edb-890d4e98d8ca/volumes"
Sep 19 11:45:08.211949 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:08.211885    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:08.475508 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:08.475409    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7" event={"ID":"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f","Type":"ContainerStarted","Data":"68fcfbc6fdadb9519cb215f3958017899cc2a29c7a213396e089e7aa383ffdf0"}
Sep 19 11:45:08.475508 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:08.475460    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7" event={"ID":"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f","Type":"ContainerStarted","Data":"aa6471114eb4547eb8fe87c0fce0b5801b3d938fec828d50aa8a97f9e8f5a011"}
Sep 19 11:45:08.497299 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:08.497234    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7" podStartSLOduration=2.4972157790000002 podStartE2EDuration="2.497215779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:45:08.494711329 +0000 UTC m=+692.694741146" watchObservedRunningTime="2026-09-19 11:45:08.497215779 +0000 UTC m=+692.697245580"
Sep 19 11:45:08.959187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:08.959150    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:09.336442 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:09.336400    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:09.342134 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:09.342103    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:09.481101 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:09.481062    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:09.482244 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:09.482221    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:09.668641 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:09.668563    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:09.831242 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:09.831210    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:09.926463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:09.926381    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:10.471443 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:10.471401    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:10.740203 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:10.740113    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:10.839892 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:10.839860    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:11.166040 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:11.166007    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:11.219302 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:11.219272    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:11.404309 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:11.404275    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:11.642247 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:11.642212    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:11.748647 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:11.748620    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:12.160265 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:12.160234    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:12.846339 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:12.846301    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:13.055728 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:13.055695    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:13.487934 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:13.487885    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:14.267439 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:14.267387    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:14.955832 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:14.955798    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:19.327311 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.327268    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"]
Sep 19 11:45:19.332594 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.332572    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.335324 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.335293    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7eb15e\"/\"kube-root-ca.crt\""
Sep 19 11:45:19.335518 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.335310    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7eb15e\"/\"openshift-service-ca.crt\""
Sep 19 11:45:19.336634 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.336602    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7eb15e\"/\"default-dockercfg-dpgm7\""
Sep 19 11:45:19.336762 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.336645    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7eb15e\"/\"simulator-w0-7eb15e-kserve-self-signed-certs\""
Sep 19 11:45:19.340057 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.340033    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"]
Sep 19 11:45:19.437360 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.437323    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad33752-6902-44e6-8d95-7954245f6787-tls-certs\") pod \"simulator-w0-7eb15e-kserve-6988dcb997-7vkc6\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") " pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.437561 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.437374    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-dshm\") pod \"simulator-w0-7eb15e-kserve-6988dcb997-7vkc6\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") " pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.437561 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.437402    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-tmp-dir\") pod \"simulator-w0-7eb15e-kserve-6988dcb997-7vkc6\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") " pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.437561 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.437419    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db88j\" (UniqueName: \"kubernetes.io/projected/aad33752-6902-44e6-8d95-7954245f6787-kube-api-access-db88j\") pod \"simulator-w0-7eb15e-kserve-6988dcb997-7vkc6\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") " pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.437561 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.437512    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-home\") pod \"simulator-w0-7eb15e-kserve-6988dcb997-7vkc6\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") " pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.437561 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.437549    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-model-cache\") pod \"simulator-w0-7eb15e-kserve-6988dcb997-7vkc6\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") " pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.538714 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.538682    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-home\") pod \"simulator-w0-7eb15e-kserve-6988dcb997-7vkc6\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") " pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.538714 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.538716    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-model-cache\") pod \"simulator-w0-7eb15e-kserve-6988dcb997-7vkc6\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") " pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.539036 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.538810    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad33752-6902-44e6-8d95-7954245f6787-tls-certs\") pod \"simulator-w0-7eb15e-kserve-6988dcb997-7vkc6\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") " pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.539036 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.538831    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-dshm\") pod \"simulator-w0-7eb15e-kserve-6988dcb997-7vkc6\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") " pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.539036 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.538985    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-tmp-dir\") pod \"simulator-w0-7eb15e-kserve-6988dcb997-7vkc6\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") " pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.539036 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.539027    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db88j\" (UniqueName: \"kubernetes.io/projected/aad33752-6902-44e6-8d95-7954245f6787-kube-api-access-db88j\") pod \"simulator-w0-7eb15e-kserve-6988dcb997-7vkc6\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") " pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.539209 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.539197    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-home\") pod \"simulator-w0-7eb15e-kserve-6988dcb997-7vkc6\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") " pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.539270 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.539247    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-model-cache\") pod \"simulator-w0-7eb15e-kserve-6988dcb997-7vkc6\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") " pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.539534 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.539509    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-tmp-dir\") pod \"simulator-w0-7eb15e-kserve-6988dcb997-7vkc6\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") " pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.541421 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.541387    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-dshm\") pod \"simulator-w0-7eb15e-kserve-6988dcb997-7vkc6\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") " pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.541833 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.541808    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad33752-6902-44e6-8d95-7954245f6787-tls-certs\") pod \"simulator-w0-7eb15e-kserve-6988dcb997-7vkc6\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") " pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.547882 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.547850    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db88j\" (UniqueName: \"kubernetes.io/projected/aad33752-6902-44e6-8d95-7954245f6787-kube-api-access-db88j\") pod \"simulator-w0-7eb15e-kserve-6988dcb997-7vkc6\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") " pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.626960 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.626857    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:19.645032 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.644997    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:19.890681 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.890585    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:19.919191 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:19.919165    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"]
Sep 19 11:45:19.921036 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:45:19.921009    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaad33752_6902_44e6_8d95_7954245f6787.slice/crio-b6c854dc0f1fab2fe01891dde491d2d4fb717fd5085d9be44649e3c1fbcec429 WatchSource:0}: Error finding container b6c854dc0f1fab2fe01891dde491d2d4fb717fd5085d9be44649e3c1fbcec429: Status 404 returned error can't find the container with id b6c854dc0f1fab2fe01891dde491d2d4fb717fd5085d9be44649e3c1fbcec429
Sep 19 11:45:20.549123 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:20.549077    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6" event={"ID":"aad33752-6902-44e6-8d95-7954245f6787","Type":"ContainerStarted","Data":"ef5de2b473feb28da2e4e30f04b5350c77bbfa3fe0e76120304c7dc36b009fab"}
Sep 19 11:45:20.549123 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:20.549129    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6" event={"ID":"aad33752-6902-44e6-8d95-7954245f6787","Type":"ContainerStarted","Data":"b6c854dc0f1fab2fe01891dde491d2d4fb717fd5085d9be44649e3c1fbcec429"}
Sep 19 11:45:20.566411 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:20.566356    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6" podStartSLOduration=1.5663361139999998 podStartE2EDuration="1.566336114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:45:20.564979149 +0000 UTC m=+704.765008961" watchObservedRunningTime="2026-09-19 11:45:20.566336114 +0000 UTC m=+704.766365918"
Sep 19 11:45:20.645380 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:20.645339    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:29.645660 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:29.645611    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:29.658727 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:29.658693    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:30.626543 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:30.626509    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:45:33.067337 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:33.067302    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:33.958041 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:33.957990    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:34.652803 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:34.652768    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:35.266836 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:35.266778    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:36.253073 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:36.252988    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:36.651567 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:36.651534    2584 generic.go:356] "Generic (PLEG): container finished" podID="a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" containerID="68fcfbc6fdadb9519cb215f3958017899cc2a29c7a213396e089e7aa383ffdf0" exitCode=0
Sep 19 11:45:36.651787 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:36.651618    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7" event={"ID":"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f","Type":"ContainerDied","Data":"68fcfbc6fdadb9519cb215f3958017899cc2a29c7a213396e089e7aa383ffdf0"}
Sep 19 11:45:36.652181 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:36.652162    2584 scope.go:117] "RemoveContainer" containerID="68fcfbc6fdadb9519cb215f3958017899cc2a29c7a213396e089e7aa383ffdf0"
Sep 19 11:45:36.653410 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:36.653381    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:45:36.653465 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:36.653454    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:45:36.653500 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:36.653489    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:45:36.928378 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:36.928291    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:37.648922 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:37.648874    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:37.659441 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:37.659404    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7" event={"ID":"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f","Type":"ContainerStarted","Data":"681dfd6cbee4fe9e7ababfa279c35656dc9aec7c1d2e01f5fa8dc3d3fc0d6d9b"}
Sep 19 11:45:38.336193 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:38.336147    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:38.337535 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:38.337502    2584 patch_prober.go:28] interesting pod/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 19 11:45:38.337776 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:38.337565    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7" podUID="a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 19 11:45:38.360156 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:38.360124    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:38.995572 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:38.995531    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:39.336611 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:39.336572    2584 patch_prober.go:28] interesting pod/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 19 11:45:39.336833 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:39.336630    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7" podUID="a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 19 11:45:39.481351 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:39.481306    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:39.769041 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:39.768954    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:40.337139 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:40.337092    2584 patch_prober.go:28] interesting pod/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 19 11:45:40.337568 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:40.337156    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7" podUID="a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 19 11:45:40.411480 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:40.411438    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:41.127324 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:41.127292    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:41.336954 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:41.336892    2584 patch_prober.go:28] interesting pod/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 19 11:45:41.337154 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:41.336972    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7" podUID="a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 19 11:45:41.854389 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:41.854350    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:41.923663 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:41.923532    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:42.019449 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:42.019419    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:42.075466 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:42.075432    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:42.336283 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:42.336251    2584 patch_prober.go:28] interesting pod/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 19 11:45:42.336486 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:42.336315    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7" podUID="a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 19 11:45:42.461422 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:42.461367    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:43.149760 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:43.149729    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:43.180791 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:43.180758    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"]
Sep 19 11:45:43.181004 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:43.180978    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7" podUID="a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" containerName="istio-proxy" containerID="cri-o://681dfd6cbee4fe9e7ababfa279c35656dc9aec7c1d2e01f5fa8dc3d3fc0d6d9b" gracePeriod=30
Sep 19 11:45:43.209074 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:43.209041    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:43.636898 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:43.636864    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:43.796051 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:43.795992    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:43.940618 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:43.940530    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:44.158200 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:44.158147    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:44.622714 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:44.622678    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:45.285271 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:45.285237    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:45.732372 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:45.732335    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:46.161624 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:46.161591    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:46.864548 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:46.864516    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:46.963363 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:46.963317    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:47.653206 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:47.653169    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:48.387837 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.387793    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:48.438968 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.438939    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:48.524657 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.524554    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istiod-ca-cert\") pod \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") "
Sep 19 11:45:48.524657 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.524592    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-token\") pod \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") "
Sep 19 11:45:48.524657 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.524612    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-envoy\") pod \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") "
Sep 19 11:45:48.525033 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.524663    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-credential-socket\") pod \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") "
Sep 19 11:45:48.525033 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.524713    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-podinfo\") pod \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") "
Sep 19 11:45:48.525033 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.524746    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-workload-certs\") pod \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") "
Sep 19 11:45:48.525033 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.524789    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hv5mb\" (UniqueName: \"kubernetes.io/projected/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-kube-api-access-hv5mb\") pod \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") "
Sep 19 11:45:48.525033 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.524819    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-data\") pod \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") "
Sep 19 11:45:48.525033 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.524843    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-workload-socket\") pod \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\" (UID: \"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f\") "
Sep 19 11:45:48.525360 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.525078    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" (UID: "a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:45:48.525360 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.525098    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" (UID: "a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:45:48.525360 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.525212    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" (UID: "a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:45:48.525360 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.525298    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-workload-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:45:48.525360 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.525318    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-credential-socket\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:45:48.525360 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.525313    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" (UID: "a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:45:48.525607 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.525488    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-data" (OuterVolumeSpecName: "istio-data") pod "a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" (UID: "a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:45:48.527135 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.527108    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-token" (OuterVolumeSpecName: "istio-token") pod "a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" (UID: "a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:45:48.527270 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.527164    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" (UID: "a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:45:48.527316 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.527266    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-kube-api-access-hv5mb" (OuterVolumeSpecName: "kube-api-access-hv5mb") pod "a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" (UID: "a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f"). InnerVolumeSpecName "kube-api-access-hv5mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:45:48.527424 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.527410    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" (UID: "a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 11:45:48.626514 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.626454    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-podinfo\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:45:48.626514 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.626511    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv5mb\" (UniqueName: \"kubernetes.io/projected/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-kube-api-access-hv5mb\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:45:48.626514 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.626522    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-data\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:45:48.626821 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.626532    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-workload-socket\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:45:48.626821 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.626540    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istiod-ca-cert\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:45:48.626821 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.626547    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-token\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:45:48.626821 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.626554    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f-istio-envoy\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:45:48.721982 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.721941    2584 generic.go:356] "Generic (PLEG): container finished" podID="a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" containerID="681dfd6cbee4fe9e7ababfa279c35656dc9aec7c1d2e01f5fa8dc3d3fc0d6d9b" exitCode=0
Sep 19 11:45:48.722152 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.721994    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"
Sep 19 11:45:48.722152 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.722010    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7" event={"ID":"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f","Type":"ContainerDied","Data":"681dfd6cbee4fe9e7ababfa279c35656dc9aec7c1d2e01f5fa8dc3d3fc0d6d9b"}
Sep 19 11:45:48.722152 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.722066    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7" event={"ID":"a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f","Type":"ContainerDied","Data":"aa6471114eb4547eb8fe87c0fce0b5801b3d938fec828d50aa8a97f9e8f5a011"}
Sep 19 11:45:48.722152 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.722089    2584 scope.go:117] "RemoveContainer" containerID="681dfd6cbee4fe9e7ababfa279c35656dc9aec7c1d2e01f5fa8dc3d3fc0d6d9b"
Sep 19 11:45:48.732289 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.732270    2584 scope.go:117] "RemoveContainer" containerID="68fcfbc6fdadb9519cb215f3958017899cc2a29c7a213396e089e7aa383ffdf0"
Sep 19 11:45:48.741406 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.741385    2584 scope.go:117] "RemoveContainer" containerID="681dfd6cbee4fe9e7ababfa279c35656dc9aec7c1d2e01f5fa8dc3d3fc0d6d9b"
Sep 19 11:45:48.741795 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:45:48.741754    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"681dfd6cbee4fe9e7ababfa279c35656dc9aec7c1d2e01f5fa8dc3d3fc0d6d9b\": container with ID starting with 681dfd6cbee4fe9e7ababfa279c35656dc9aec7c1d2e01f5fa8dc3d3fc0d6d9b not found: ID does not exist" containerID="681dfd6cbee4fe9e7ababfa279c35656dc9aec7c1d2e01f5fa8dc3d3fc0d6d9b"
Sep 19 11:45:48.741795 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.741784    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"681dfd6cbee4fe9e7ababfa279c35656dc9aec7c1d2e01f5fa8dc3d3fc0d6d9b"} err="failed to get container status \"681dfd6cbee4fe9e7ababfa279c35656dc9aec7c1d2e01f5fa8dc3d3fc0d6d9b\": rpc error: code = NotFound desc = could not find container \"681dfd6cbee4fe9e7ababfa279c35656dc9aec7c1d2e01f5fa8dc3d3fc0d6d9b\": container with ID starting with 681dfd6cbee4fe9e7ababfa279c35656dc9aec7c1d2e01f5fa8dc3d3fc0d6d9b not found: ID does not exist"
Sep 19 11:45:48.741894 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.741803    2584 scope.go:117] "RemoveContainer" containerID="68fcfbc6fdadb9519cb215f3958017899cc2a29c7a213396e089e7aa383ffdf0"
Sep 19 11:45:48.742175 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:45:48.742151    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68fcfbc6fdadb9519cb215f3958017899cc2a29c7a213396e089e7aa383ffdf0\": container with ID starting with 68fcfbc6fdadb9519cb215f3958017899cc2a29c7a213396e089e7aa383ffdf0 not found: ID does not exist" containerID="68fcfbc6fdadb9519cb215f3958017899cc2a29c7a213396e089e7aa383ffdf0"
Sep 19 11:45:48.742271 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.742187    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68fcfbc6fdadb9519cb215f3958017899cc2a29c7a213396e089e7aa383ffdf0"} err="failed to get container status \"68fcfbc6fdadb9519cb215f3958017899cc2a29c7a213396e089e7aa383ffdf0\": rpc error: code = NotFound desc = could not find container \"68fcfbc6fdadb9519cb215f3958017899cc2a29c7a213396e089e7aa383ffdf0\": container with ID starting with 68fcfbc6fdadb9519cb215f3958017899cc2a29c7a213396e089e7aa383ffdf0 not found: ID does not exist"
Sep 19 11:45:48.744661 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.744640    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"]
Sep 19 11:45:48.748564 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:48.748542    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5deb747f-openshift-default-596dd4dfd8-vw4x7"]
Sep 19 11:45:49.075838 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:49.075805    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:49.754392 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:49.754355    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:49.906317 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:49.906275    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" path="/var/lib/kubelet/pods/a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f/volumes"
Sep 19 11:45:50.520829 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:50.520796    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:51.160608 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:51.160572    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:51.259405 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:51.259375    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:51.362933 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:51.362886    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:51.614867 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:51.614836    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:51.714740 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:51.714694    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:53.930248 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:53.930184    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"]
Sep 19 11:45:53.930841 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:53.930818    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" containerName="istio-proxy"
Sep 19 11:45:53.930956 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:53.930845    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" containerName="istio-proxy"
Sep 19 11:45:53.931029 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:53.931014    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" containerName="istio-proxy"
Sep 19 11:45:53.934216 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:53.934193    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:53.937045 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:53.937016    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7a524f\"/\"kube-root-ca.crt\""
Sep 19 11:45:53.937045 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:53.937016    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7a524f\"/\"openshift-service-ca.crt\""
Sep 19 11:45:53.937236 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:53.937164    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7a524f\"/\"autores-model-7a524f-kserve-self-signed-certs\""
Sep 19 11:45:53.938269 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:53.938233    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7a524f\"/\"default-dockercfg-nmhv7\""
Sep 19 11:45:53.940832 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:53.940806    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"]
Sep 19 11:45:54.078767 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.078731    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-home\") pod \"autores-model-7a524f-kserve-856ff668fb-qkg8l\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") " pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:54.078975 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.078776    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-tmp-dir\") pod \"autores-model-7a524f-kserve-856ff668fb-qkg8l\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") " pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:54.078975 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.078904    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-tls-certs\") pod \"autores-model-7a524f-kserve-856ff668fb-qkg8l\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") " pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:54.079064 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.078985    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-dshm\") pod \"autores-model-7a524f-kserve-856ff668fb-qkg8l\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") " pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:54.079064 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.079018    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tgtk\" (UniqueName: \"kubernetes.io/projected/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-kube-api-access-9tgtk\") pod \"autores-model-7a524f-kserve-856ff668fb-qkg8l\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") " pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:54.079064 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.079052    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-model-cache\") pod \"autores-model-7a524f-kserve-856ff668fb-qkg8l\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") " pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:54.180368 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.180261    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-home\") pod \"autores-model-7a524f-kserve-856ff668fb-qkg8l\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") " pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:54.180368 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.180307    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-tmp-dir\") pod \"autores-model-7a524f-kserve-856ff668fb-qkg8l\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") " pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:54.180368 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.180358    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-tls-certs\") pod \"autores-model-7a524f-kserve-856ff668fb-qkg8l\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") " pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:54.180660 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.180385    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-dshm\") pod \"autores-model-7a524f-kserve-856ff668fb-qkg8l\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") " pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:54.180660 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.180412    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tgtk\" (UniqueName: \"kubernetes.io/projected/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-kube-api-access-9tgtk\") pod \"autores-model-7a524f-kserve-856ff668fb-qkg8l\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") " pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:54.180660 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.180439    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-model-cache\") pod \"autores-model-7a524f-kserve-856ff668fb-qkg8l\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") " pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:54.180806 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.180732    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-home\") pod \"autores-model-7a524f-kserve-856ff668fb-qkg8l\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") " pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:54.180806 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.180757    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-tmp-dir\") pod \"autores-model-7a524f-kserve-856ff668fb-qkg8l\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") " pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:54.180959 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.180814    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-model-cache\") pod \"autores-model-7a524f-kserve-856ff668fb-qkg8l\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") " pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:54.182864 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.182841    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-dshm\") pod \"autores-model-7a524f-kserve-856ff668fb-qkg8l\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") " pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:54.183075 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.183059    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-tls-certs\") pod \"autores-model-7a524f-kserve-856ff668fb-qkg8l\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") " pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:54.188835 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.188807    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tgtk\" (UniqueName: \"kubernetes.io/projected/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-kube-api-access-9tgtk\") pod \"autores-model-7a524f-kserve-856ff668fb-qkg8l\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") " pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:54.244630 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.244586    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:54.435868 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.435779    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:45:54.523520 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.523490    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"]
Sep 19 11:45:54.525343 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:45:54.525312    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b107b69_40e1_4d5c_9b97_6d9cbfbcf608.slice/crio-5419bb01f90c96899587116302114fcd65d96dcdd364d16a9d791c597dcb77b0 WatchSource:0}: Error finding container 5419bb01f90c96899587116302114fcd65d96dcdd364d16a9d791c597dcb77b0: Status 404 returned error can't find the container with id 5419bb01f90c96899587116302114fcd65d96dcdd364d16a9d791c597dcb77b0
Sep 19 11:45:54.756954 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.756815    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l" event={"ID":"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608","Type":"ContainerStarted","Data":"1db2f4fb9e745d7b1a8b906d75d9d3b6e55cfc896612e9954b7bf4c41fd8dc51"}
Sep 19 11:45:54.756954 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.756860    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l" event={"ID":"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608","Type":"ContainerStarted","Data":"5419bb01f90c96899587116302114fcd65d96dcdd364d16a9d791c597dcb77b0"}
Sep 19 11:45:54.777815 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:54.777744    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l" podStartSLOduration=1.777721383 podStartE2EDuration="1.777721383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:45:54.773479939 +0000 UTC m=+738.973509743" watchObservedRunningTime="2026-09-19 11:45:54.777721383 +0000 UTC m=+738.977751187"
Sep 19 11:45:55.245259 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:55.245215    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:45:59.465423 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:45:59.465385    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:04.245046 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:04.245003    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:46:04.258187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:04.258105    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:46:04.821484 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:04.821452    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:46:12.079061 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:12.079028    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:13.544933 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:13.544874    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:14.230832 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:14.230797    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:14.953341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:14.953310    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:15.944549 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:15.944515    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:16.638275 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:16.638245    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:17.602560 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:17.602524    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:18.295488 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:18.295456    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:19.163773 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:19.163740    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:19.847320 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:19.847260    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:20.507262 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:20.507225    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:21.360661 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:21.360622    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:21.994971 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:21.994934    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:26.650178 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:26.650142    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:27.434182 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:27.434147    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:28.068761 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:28.068727    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:28.792212 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:28.792181    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:29.340272 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:29.340231    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:29.404110 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:29.404081    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:29.504126 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:29.504091    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:29.556606 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:29.556577    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:29.618402 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:29.618311    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:29.846370 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:29.846340    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:29.944367 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:29.944273    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:30.537453 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:30.537420    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:30.618621 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:30.618585    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:30.684531 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:30.684493    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:31.355457 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:31.355420    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:31.998830 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:31.998796    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:32.098903 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:32.098870    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:32.752839 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:32.752804    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:33.380023 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:33.379991    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:33.999803 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:33.999762    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:36.253456 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:36.253416    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:36.896882 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:36.896835    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"]
Sep 19 11:46:36.897394 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:36.897357    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798" podUID="1f5aa702-65ad-4c79-ae23-57542b7a195e" containerName="maas-api" containerID="cri-o://22b02028fb7d24984449a6f54691794f01c2c989328fafe0860c3fc9c6c010a2" gracePeriod=30
Sep 19 11:46:36.929668 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:36.929515    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:37.027411 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.027383    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:37.109392 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.109356    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:37.160337 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.160273    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"
Sep 19 11:46:37.164607 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.164582    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:37.304422 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.304377    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtcmz\" (UniqueName: \"kubernetes.io/projected/1f5aa702-65ad-4c79-ae23-57542b7a195e-kube-api-access-vtcmz\") pod \"1f5aa702-65ad-4c79-ae23-57542b7a195e\" (UID: \"1f5aa702-65ad-4c79-ae23-57542b7a195e\") "
Sep 19 11:46:37.304900 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.304441    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f5aa702-65ad-4c79-ae23-57542b7a195e-maas-api-tls\") pod \"1f5aa702-65ad-4c79-ae23-57542b7a195e\" (UID: \"1f5aa702-65ad-4c79-ae23-57542b7a195e\") "
Sep 19 11:46:37.304900 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.304499    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f5aa702-65ad-4c79-ae23-57542b7a195e-metrics-certs\") pod \"1f5aa702-65ad-4c79-ae23-57542b7a195e\" (UID: \"1f5aa702-65ad-4c79-ae23-57542b7a195e\") "
Sep 19 11:46:37.304900 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.304551    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5aa702-65ad-4c79-ae23-57542b7a195e-openshift-service-ca\") pod \"1f5aa702-65ad-4c79-ae23-57542b7a195e\" (UID: \"1f5aa702-65ad-4c79-ae23-57542b7a195e\") "
Sep 19 11:46:37.305085 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.304953    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f5aa702-65ad-4c79-ae23-57542b7a195e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1f5aa702-65ad-4c79-ae23-57542b7a195e" (UID: "1f5aa702-65ad-4c79-ae23-57542b7a195e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:46:37.306853 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.306822    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f5aa702-65ad-4c79-ae23-57542b7a195e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1f5aa702-65ad-4c79-ae23-57542b7a195e" (UID: "1f5aa702-65ad-4c79-ae23-57542b7a195e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:46:37.307324 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.307305    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f5aa702-65ad-4c79-ae23-57542b7a195e-kube-api-access-vtcmz" (OuterVolumeSpecName: "kube-api-access-vtcmz") pod "1f5aa702-65ad-4c79-ae23-57542b7a195e" (UID: "1f5aa702-65ad-4c79-ae23-57542b7a195e"). InnerVolumeSpecName "kube-api-access-vtcmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:46:37.307324 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.307309    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f5aa702-65ad-4c79-ae23-57542b7a195e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1f5aa702-65ad-4c79-ae23-57542b7a195e" (UID: "1f5aa702-65ad-4c79-ae23-57542b7a195e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:46:37.406161 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.406114    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtcmz\" (UniqueName: \"kubernetes.io/projected/1f5aa702-65ad-4c79-ae23-57542b7a195e-kube-api-access-vtcmz\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:46:37.406161 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.406149    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f5aa702-65ad-4c79-ae23-57542b7a195e-maas-api-tls\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:46:37.406161 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.406158    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f5aa702-65ad-4c79-ae23-57542b7a195e-metrics-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:46:37.406161 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.406166    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5aa702-65ad-4c79-ae23-57542b7a195e-openshift-service-ca\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:46:37.454290 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.454199    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:37.554731 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.554698    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:37.990008 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.989969    2584 generic.go:356] "Generic (PLEG): container finished" podID="1f5aa702-65ad-4c79-ae23-57542b7a195e" containerID="22b02028fb7d24984449a6f54691794f01c2c989328fafe0860c3fc9c6c010a2" exitCode=0
Sep 19 11:46:37.990213 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.990008    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798" event={"ID":"1f5aa702-65ad-4c79-ae23-57542b7a195e","Type":"ContainerDied","Data":"22b02028fb7d24984449a6f54691794f01c2c989328fafe0860c3fc9c6c010a2"}
Sep 19 11:46:37.990213 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.990084    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798" event={"ID":"1f5aa702-65ad-4c79-ae23-57542b7a195e","Type":"ContainerDied","Data":"00ca1d0c93d5c9c21dbdf2bf2fe49888bb39317f04fe137fa69670038d7edef3"}
Sep 19 11:46:37.990213 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.990021    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"
Sep 19 11:46:37.990213 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:37.990117    2584 scope.go:117] "RemoveContainer" containerID="22b02028fb7d24984449a6f54691794f01c2c989328fafe0860c3fc9c6c010a2"
Sep 19 11:46:38.000273 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:38.000248    2584 scope.go:117] "RemoveContainer" containerID="22b02028fb7d24984449a6f54691794f01c2c989328fafe0860c3fc9c6c010a2"
Sep 19 11:46:38.000634 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:46:38.000612    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b02028fb7d24984449a6f54691794f01c2c989328fafe0860c3fc9c6c010a2\": container with ID starting with 22b02028fb7d24984449a6f54691794f01c2c989328fafe0860c3fc9c6c010a2 not found: ID does not exist" containerID="22b02028fb7d24984449a6f54691794f01c2c989328fafe0860c3fc9c6c010a2"
Sep 19 11:46:38.000704 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:38.000644    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b02028fb7d24984449a6f54691794f01c2c989328fafe0860c3fc9c6c010a2"} err="failed to get container status \"22b02028fb7d24984449a6f54691794f01c2c989328fafe0860c3fc9c6c010a2\": rpc error: code = NotFound desc = could not find container \"22b02028fb7d24984449a6f54691794f01c2c989328fafe0860c3fc9c6c010a2\": container with ID starting with 22b02028fb7d24984449a6f54691794f01c2c989328fafe0860c3fc9c6c010a2 not found: ID does not exist"
Sep 19 11:46:38.008311 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:38.008281    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"]
Sep 19 11:46:38.012873 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:38.012844    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7a524f-8479d97798-bc798"]
Sep 19 11:46:38.222130 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:38.222043    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:38.324806 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:38.324768    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:38.422100 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:38.422066    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:38.523042 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:38.522955    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:38.898112 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:38.898073    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:39.157290 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:39.157183    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:39.863155 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:39.863105    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:39.903845 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:39.903808    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f5aa702-65ad-4c79-ae23-57542b7a195e" path="/var/lib/kubelet/pods/1f5aa702-65ad-4c79-ae23-57542b7a195e/volumes"
Sep 19 11:46:40.605026 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:40.604992    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:42.435455 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:42.435414    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"]
Sep 19 11:46:42.478056 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:42.478024    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:46:57.456245 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.456148    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l" podUID="0b107b69-40e1-4d5c-9b97-6d9cbfbcf608" containerName="main" containerID="cri-o://1db2f4fb9e745d7b1a8b906d75d9d3b6e55cfc896612e9954b7bf4c41fd8dc51" gracePeriod=45
Sep 19 11:46:57.718727 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.718660    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:46:57.797074 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.797043    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-dshm\") pod \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") "
Sep 19 11:46:57.797280 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.797095    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-model-cache\") pod \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") "
Sep 19 11:46:57.797280 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.797172    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-tls-certs\") pod \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") "
Sep 19 11:46:57.797280 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.797202    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-home\") pod \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") "
Sep 19 11:46:57.797280 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.797249    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tgtk\" (UniqueName: \"kubernetes.io/projected/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-kube-api-access-9tgtk\") pod \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") "
Sep 19 11:46:57.797482 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.797310    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-tmp-dir\") pod \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\" (UID: \"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608\") "
Sep 19 11:46:57.797482 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.797426    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b107b69-40e1-4d5c-9b97-6d9cbfbcf608" (UID: "0b107b69-40e1-4d5c-9b97-6d9cbfbcf608"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:46:57.797683 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.797633    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b107b69-40e1-4d5c-9b97-6d9cbfbcf608" (UID: "0b107b69-40e1-4d5c-9b97-6d9cbfbcf608"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:46:57.797683 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.797656    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-home" (OuterVolumeSpecName: "home") pod "0b107b69-40e1-4d5c-9b97-6d9cbfbcf608" (UID: "0b107b69-40e1-4d5c-9b97-6d9cbfbcf608"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:46:57.797861 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.797667    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-model-cache\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:46:57.799721 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.799692    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-dshm" (OuterVolumeSpecName: "dshm") pod "0b107b69-40e1-4d5c-9b97-6d9cbfbcf608" (UID: "0b107b69-40e1-4d5c-9b97-6d9cbfbcf608"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:46:57.799971 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.799942    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b107b69-40e1-4d5c-9b97-6d9cbfbcf608" (UID: "0b107b69-40e1-4d5c-9b97-6d9cbfbcf608"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:46:57.800053 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.800015    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-kube-api-access-9tgtk" (OuterVolumeSpecName: "kube-api-access-9tgtk") pod "0b107b69-40e1-4d5c-9b97-6d9cbfbcf608" (UID: "0b107b69-40e1-4d5c-9b97-6d9cbfbcf608"). InnerVolumeSpecName "kube-api-access-9tgtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:46:57.899115 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.899074    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-tls-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:46:57.899115 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.899102    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-home\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:46:57.899115 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.899111    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tgtk\" (UniqueName: \"kubernetes.io/projected/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-kube-api-access-9tgtk\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:46:57.899115 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.899120    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-tmp-dir\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:46:57.899115 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:57.899130    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608-dshm\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:46:58.116603 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:58.116564    2584 generic.go:356] "Generic (PLEG): container finished" podID="0b107b69-40e1-4d5c-9b97-6d9cbfbcf608" containerID="1db2f4fb9e745d7b1a8b906d75d9d3b6e55cfc896612e9954b7bf4c41fd8dc51" exitCode=0
Sep 19 11:46:58.116801 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:58.116614    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"
Sep 19 11:46:58.116801 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:58.116666    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l" event={"ID":"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608","Type":"ContainerDied","Data":"1db2f4fb9e745d7b1a8b906d75d9d3b6e55cfc896612e9954b7bf4c41fd8dc51"}
Sep 19 11:46:58.116801 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:58.116703    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l" event={"ID":"0b107b69-40e1-4d5c-9b97-6d9cbfbcf608","Type":"ContainerDied","Data":"5419bb01f90c96899587116302114fcd65d96dcdd364d16a9d791c597dcb77b0"}
Sep 19 11:46:58.116801 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:58.116729    2584 scope.go:117] "RemoveContainer" containerID="1db2f4fb9e745d7b1a8b906d75d9d3b6e55cfc896612e9954b7bf4c41fd8dc51"
Sep 19 11:46:58.126382 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:58.126358    2584 scope.go:117] "RemoveContainer" containerID="1db2f4fb9e745d7b1a8b906d75d9d3b6e55cfc896612e9954b7bf4c41fd8dc51"
Sep 19 11:46:58.126698 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:46:58.126673    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db2f4fb9e745d7b1a8b906d75d9d3b6e55cfc896612e9954b7bf4c41fd8dc51\": container with ID starting with 1db2f4fb9e745d7b1a8b906d75d9d3b6e55cfc896612e9954b7bf4c41fd8dc51 not found: ID does not exist" containerID="1db2f4fb9e745d7b1a8b906d75d9d3b6e55cfc896612e9954b7bf4c41fd8dc51"
Sep 19 11:46:58.126815 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:58.126705    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db2f4fb9e745d7b1a8b906d75d9d3b6e55cfc896612e9954b7bf4c41fd8dc51"} err="failed to get container status \"1db2f4fb9e745d7b1a8b906d75d9d3b6e55cfc896612e9954b7bf4c41fd8dc51\": rpc error: code = NotFound desc = could not find container \"1db2f4fb9e745d7b1a8b906d75d9d3b6e55cfc896612e9954b7bf4c41fd8dc51\": container with ID starting with 1db2f4fb9e745d7b1a8b906d75d9d3b6e55cfc896612e9954b7bf4c41fd8dc51 not found: ID does not exist"
Sep 19 11:46:58.132740 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:58.132707    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"]
Sep 19 11:46:58.136172 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:58.136129    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-7a524f/autores-model-7a524f-kserve-856ff668fb-qkg8l"]
Sep 19 11:46:59.904192 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:46:59.904157    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b107b69-40e1-4d5c-9b97-6d9cbfbcf608" path="/var/lib/kubelet/pods/0b107b69-40e1-4d5c-9b97-6d9cbfbcf608/volumes"
Sep 19 11:47:03.224209 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.224168    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"]
Sep 19 11:47:03.224719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.224640    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b107b69-40e1-4d5c-9b97-6d9cbfbcf608" containerName="main"
Sep 19 11:47:03.224719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.224656    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b107b69-40e1-4d5c-9b97-6d9cbfbcf608" containerName="main"
Sep 19 11:47:03.224719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.224673    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" containerName="istio-proxy"
Sep 19 11:47:03.224719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.224678    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" containerName="istio-proxy"
Sep 19 11:47:03.224719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.224687    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f5aa702-65ad-4c79-ae23-57542b7a195e" containerName="maas-api"
Sep 19 11:47:03.224719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.224695    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f5aa702-65ad-4c79-ae23-57542b7a195e" containerName="maas-api"
Sep 19 11:47:03.225091 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.224772    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a39aa8a3-2686-4cc1-a684-82b2ad9a5f8f" containerName="istio-proxy"
Sep 19 11:47:03.225091 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.224780    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b107b69-40e1-4d5c-9b97-6d9cbfbcf608" containerName="main"
Sep 19 11:47:03.225091 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.224787    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f5aa702-65ad-4c79-ae23-57542b7a195e" containerName="maas-api"
Sep 19 11:47:03.229454 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.229422    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.233436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.233399    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-333f49\"/\"openshift-service-ca.crt\""
Sep 19 11:47:03.233609 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.233453    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-333f49\"/\"kube-root-ca.crt\""
Sep 19 11:47:03.233609 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.233402    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-333f49\"/\"test-model-333f49-kserve-self-signed-certs\""
Sep 19 11:47:03.233843 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.233820    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-333f49\"/\"default-dockercfg-gxmhd\""
Sep 19 11:47:03.236873 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.236837    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"]
Sep 19 11:47:03.241966 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.241935    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:03.353437 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.353398    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-home\") pod \"test-model-333f49-kserve-5c9647f9c4-6hfht\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") " pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.353634 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.353469    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-model-cache\") pod \"test-model-333f49-kserve-5c9647f9c4-6hfht\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") " pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.353634 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.353495    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c151ef3-0db2-4693-bb09-1ba0e66c3968-tls-certs\") pod \"test-model-333f49-kserve-5c9647f9c4-6hfht\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") " pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.353634 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.353518    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncm6z\" (UniqueName: \"kubernetes.io/projected/4c151ef3-0db2-4693-bb09-1ba0e66c3968-kube-api-access-ncm6z\") pod \"test-model-333f49-kserve-5c9647f9c4-6hfht\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") " pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.353634 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.353549    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-tmp-dir\") pod \"test-model-333f49-kserve-5c9647f9c4-6hfht\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") " pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.353634 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.353613    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-dshm\") pod \"test-model-333f49-kserve-5c9647f9c4-6hfht\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") " pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.454880 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.454838    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-home\") pod \"test-model-333f49-kserve-5c9647f9c4-6hfht\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") " pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.455117 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.454897    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-model-cache\") pod \"test-model-333f49-kserve-5c9647f9c4-6hfht\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") " pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.455117 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.454938    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c151ef3-0db2-4693-bb09-1ba0e66c3968-tls-certs\") pod \"test-model-333f49-kserve-5c9647f9c4-6hfht\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") " pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.455117 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.454961    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncm6z\" (UniqueName: \"kubernetes.io/projected/4c151ef3-0db2-4693-bb09-1ba0e66c3968-kube-api-access-ncm6z\") pod \"test-model-333f49-kserve-5c9647f9c4-6hfht\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") " pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.455117 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.454991    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-tmp-dir\") pod \"test-model-333f49-kserve-5c9647f9c4-6hfht\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") " pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.455117 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.455023    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-dshm\") pod \"test-model-333f49-kserve-5c9647f9c4-6hfht\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") " pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.455397 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.455381    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-model-cache\") pod \"test-model-333f49-kserve-5c9647f9c4-6hfht\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") " pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.455451 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.455405    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-home\") pod \"test-model-333f49-kserve-5c9647f9c4-6hfht\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") " pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.455490 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.455456    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-tmp-dir\") pod \"test-model-333f49-kserve-5c9647f9c4-6hfht\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") " pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.457557 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.457532    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-dshm\") pod \"test-model-333f49-kserve-5c9647f9c4-6hfht\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") " pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.457868 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.457850    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c151ef3-0db2-4693-bb09-1ba0e66c3968-tls-certs\") pod \"test-model-333f49-kserve-5c9647f9c4-6hfht\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") " pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.463286 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.463255    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncm6z\" (UniqueName: \"kubernetes.io/projected/4c151ef3-0db2-4693-bb09-1ba0e66c3968-kube-api-access-ncm6z\") pod \"test-model-333f49-kserve-5c9647f9c4-6hfht\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") " pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.541077 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.540977    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:03.821534 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:03.821504    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"]
Sep 19 11:47:03.823486 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:47:03.823462    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c151ef3_0db2_4693_bb09_1ba0e66c3968.slice/crio-b300bc288e79d8db41f38b96b49643cc65b0e9d646ea9323e83c86921ec47be8 WatchSource:0}: Error finding container b300bc288e79d8db41f38b96b49643cc65b0e9d646ea9323e83c86921ec47be8: Status 404 returned error can't find the container with id b300bc288e79d8db41f38b96b49643cc65b0e9d646ea9323e83c86921ec47be8
Sep 19 11:47:04.151271 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:04.151168    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht" event={"ID":"4c151ef3-0db2-4693-bb09-1ba0e66c3968","Type":"ContainerStarted","Data":"bbe782c7f108ff72f6d7c76035ba49b084bd63b1afd6042b4aa34cfa288bfe17"}
Sep 19 11:47:04.151271 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:04.151215    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht" event={"ID":"4c151ef3-0db2-4693-bb09-1ba0e66c3968","Type":"ContainerStarted","Data":"b300bc288e79d8db41f38b96b49643cc65b0e9d646ea9323e83c86921ec47be8"}
Sep 19 11:47:04.170905 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:04.170852    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht" podStartSLOduration=1.1708353009999999 podStartE2EDuration="1.170835301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:47:04.168561134 +0000 UTC m=+808.368590938" watchObservedRunningTime="2026-09-19 11:47:04.170835301 +0000 UTC m=+808.370865103"
Sep 19 11:47:04.541589 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:04.541495    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:04.817684 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:04.817650    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:10.490083 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:10.490044    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:10.591464 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:10.591356    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:10.682906 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:10.682873    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:10.788148 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:10.788046    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:11.583709 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:11.583676    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:12.364025 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:12.363983    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:13.152201 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:13.152150    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:13.541893 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:13.541794    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:13.555309 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:13.555277    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:13.788775 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:13.788732    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:14.218640 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:14.218609    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:47:14.552027 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:14.551945    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:15.619632 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:15.619597    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:18.235352 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:18.235321    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:19.248888 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:19.248858    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:19.474660 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:19.474625    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:19.900213 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:19.900175    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:20.188326 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:20.188219    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:20.251487 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:20.251445    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:21.103686 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.103644    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"]
Sep 19 11:47:21.103953 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.103893    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f" podUID="ba69ec7a-0643-4b63-9f60-093bd4a10f26" containerName="maas-api" containerID="cri-o://0dc55cda4977d80155a7fd168eb80844b28ed08bd98c1a537c511e7df9ba7fa0" gracePeriod=30
Sep 19 11:47:21.130611 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.130576    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:21.208996 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.208956    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:21.250898 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.250865    2584 generic.go:356] "Generic (PLEG): container finished" podID="ba69ec7a-0643-4b63-9f60-093bd4a10f26" containerID="0dc55cda4977d80155a7fd168eb80844b28ed08bd98c1a537c511e7df9ba7fa0" exitCode=0
Sep 19 11:47:21.251111 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.250957    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f" event={"ID":"ba69ec7a-0643-4b63-9f60-093bd4a10f26","Type":"ContainerDied","Data":"0dc55cda4977d80155a7fd168eb80844b28ed08bd98c1a537c511e7df9ba7fa0"}
Sep 19 11:47:21.267164 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.267125    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:21.331318 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.331285    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:21.376119 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.376041    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"
Sep 19 11:47:21.386064 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.386037    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:21.533875 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.533836    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba69ec7a-0643-4b63-9f60-093bd4a10f26-maas-api-tls\") pod \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\" (UID: \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\") "
Sep 19 11:47:21.533875 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.533889    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba69ec7a-0643-4b63-9f60-093bd4a10f26-openshift-service-ca\") pod \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\" (UID: \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\") "
Sep 19 11:47:21.534197 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.533958    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtzxd\" (UniqueName: \"kubernetes.io/projected/ba69ec7a-0643-4b63-9f60-093bd4a10f26-kube-api-access-gtzxd\") pod \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\" (UID: \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\") "
Sep 19 11:47:21.534197 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.533975    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba69ec7a-0643-4b63-9f60-093bd4a10f26-metrics-certs\") pod \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\" (UID: \"ba69ec7a-0643-4b63-9f60-093bd4a10f26\") "
Sep 19 11:47:21.534375 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.534349    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba69ec7a-0643-4b63-9f60-093bd4a10f26-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ba69ec7a-0643-4b63-9f60-093bd4a10f26" (UID: "ba69ec7a-0643-4b63-9f60-093bd4a10f26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:47:21.536393 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.536361    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba69ec7a-0643-4b63-9f60-093bd4a10f26-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ba69ec7a-0643-4b63-9f60-093bd4a10f26" (UID: "ba69ec7a-0643-4b63-9f60-093bd4a10f26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:47:21.536393 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.536377    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba69ec7a-0643-4b63-9f60-093bd4a10f26-kube-api-access-gtzxd" (OuterVolumeSpecName: "kube-api-access-gtzxd") pod "ba69ec7a-0643-4b63-9f60-093bd4a10f26" (UID: "ba69ec7a-0643-4b63-9f60-093bd4a10f26"). InnerVolumeSpecName "kube-api-access-gtzxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:47:21.536538 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.536378    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba69ec7a-0643-4b63-9f60-093bd4a10f26-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ba69ec7a-0643-4b63-9f60-093bd4a10f26" (UID: "ba69ec7a-0643-4b63-9f60-093bd4a10f26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:47:21.634809 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.634721    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba69ec7a-0643-4b63-9f60-093bd4a10f26-maas-api-tls\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:47:21.634809 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.634754    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba69ec7a-0643-4b63-9f60-093bd4a10f26-openshift-service-ca\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:47:21.634809 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.634764    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtzxd\" (UniqueName: \"kubernetes.io/projected/ba69ec7a-0643-4b63-9f60-093bd4a10f26-kube-api-access-gtzxd\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:47:21.634809 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.634774    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba69ec7a-0643-4b63-9f60-093bd4a10f26-metrics-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:47:21.667291 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.667255    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:21.778224 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:21.778191    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:22.258293 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:22.258184    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"
Sep 19 11:47:22.258293 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:22.258191    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f" event={"ID":"ba69ec7a-0643-4b63-9f60-093bd4a10f26","Type":"ContainerDied","Data":"c1b8c09e1c1bd274591d324643e814fa8db184886d0297a48b2092a25f2eef40"}
Sep 19 11:47:22.258588 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:22.258321    2584 scope.go:117] "RemoveContainer" containerID="0dc55cda4977d80155a7fd168eb80844b28ed08bd98c1a537c511e7df9ba7fa0"
Sep 19 11:47:22.274303 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:22.274261    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"]
Sep 19 11:47:22.277999 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:22.277965    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-37daed-669bfcd664-mdp6f"]
Sep 19 11:47:22.556273 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:22.556189    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:23.267876 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:23.267841    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:23.360408 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:23.360373    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:23.460114 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:23.460076    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:23.904254 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:23.904211    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba69ec7a-0643-4b63-9f60-093bd4a10f26" path="/var/lib/kubelet/pods/ba69ec7a-0643-4b63-9f60-093bd4a10f26/volumes"
Sep 19 11:47:24.089007 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:24.088972    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:24.917659 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:24.917609    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"]
Sep 19 11:47:24.918288 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:24.918266    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba69ec7a-0643-4b63-9f60-093bd4a10f26" containerName="maas-api"
Sep 19 11:47:24.918337 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:24.918293    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba69ec7a-0643-4b63-9f60-093bd4a10f26" containerName="maas-api"
Sep 19 11:47:24.918417 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:24.918405    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba69ec7a-0643-4b63-9f60-093bd4a10f26" containerName="maas-api"
Sep 19 11:47:24.922902 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:24.922878    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"
Sep 19 11:47:24.925568 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:24.925534    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-a527d4-serving-cert\""
Sep 19 11:47:24.930990 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:24.930954    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"]
Sep 19 11:47:24.949270 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:24.949240    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:24.964658 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:24.964621    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10272086-df57-4b14-bca8-2aa5b04f6ac1-maas-api-tls\") pod \"maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q\" (UID: \"10272086-df57-4b14-bca8-2aa5b04f6ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"
Sep 19 11:47:24.964892 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:24.964865    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10272086-df57-4b14-bca8-2aa5b04f6ac1-openshift-service-ca\") pod \"maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q\" (UID: \"10272086-df57-4b14-bca8-2aa5b04f6ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"
Sep 19 11:47:24.965074 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:24.965058    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10272086-df57-4b14-bca8-2aa5b04f6ac1-metrics-certs\") pod \"maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q\" (UID: \"10272086-df57-4b14-bca8-2aa5b04f6ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"
Sep 19 11:47:24.965146 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:24.965094    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5xnd\" (UniqueName: \"kubernetes.io/projected/10272086-df57-4b14-bca8-2aa5b04f6ac1-kube-api-access-m5xnd\") pod \"maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q\" (UID: \"10272086-df57-4b14-bca8-2aa5b04f6ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"
Sep 19 11:47:25.052110 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:25.052073    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:25.066389 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:25.066355    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10272086-df57-4b14-bca8-2aa5b04f6ac1-openshift-service-ca\") pod \"maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q\" (UID: \"10272086-df57-4b14-bca8-2aa5b04f6ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"
Sep 19 11:47:25.066566 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:25.066435    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10272086-df57-4b14-bca8-2aa5b04f6ac1-metrics-certs\") pod \"maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q\" (UID: \"10272086-df57-4b14-bca8-2aa5b04f6ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"
Sep 19 11:47:25.066566 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:25.066457    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5xnd\" (UniqueName: \"kubernetes.io/projected/10272086-df57-4b14-bca8-2aa5b04f6ac1-kube-api-access-m5xnd\") pod \"maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q\" (UID: \"10272086-df57-4b14-bca8-2aa5b04f6ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"
Sep 19 11:47:25.066566 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:25.066483    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10272086-df57-4b14-bca8-2aa5b04f6ac1-maas-api-tls\") pod \"maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q\" (UID: \"10272086-df57-4b14-bca8-2aa5b04f6ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"
Sep 19 11:47:25.066732 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:47:25.066589    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-serving-cert: secret "maas-api-e2e-explicit-a527d4-serving-cert" not found
Sep 19 11:47:25.066732 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:47:25.066654    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10272086-df57-4b14-bca8-2aa5b04f6ac1-maas-api-tls podName:10272086-df57-4b14-bca8-2aa5b04f6ac1 nodeName:}" failed. No retries permitted until 2026-09-19 11:47:25.566638886 +0000 UTC m=+829.766668667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/10272086-df57-4b14-bca8-2aa5b04f6ac1-maas-api-tls") pod "maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q" (UID: "10272086-df57-4b14-bca8-2aa5b04f6ac1") : secret "maas-api-e2e-explicit-a527d4-serving-cert" not found
Sep 19 11:47:25.067121 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:25.067097    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10272086-df57-4b14-bca8-2aa5b04f6ac1-openshift-service-ca\") pod \"maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q\" (UID: \"10272086-df57-4b14-bca8-2aa5b04f6ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"
Sep 19 11:47:25.069206 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:25.069183    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10272086-df57-4b14-bca8-2aa5b04f6ac1-metrics-certs\") pod \"maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q\" (UID: \"10272086-df57-4b14-bca8-2aa5b04f6ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"
Sep 19 11:47:25.075290 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:25.075260    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5xnd\" (UniqueName: \"kubernetes.io/projected/10272086-df57-4b14-bca8-2aa5b04f6ac1-kube-api-access-m5xnd\") pod \"maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q\" (UID: \"10272086-df57-4b14-bca8-2aa5b04f6ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"
Sep 19 11:47:25.392018 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:25.391983    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:25.484185 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:25.484156    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:25.572371 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:25.572331    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10272086-df57-4b14-bca8-2aa5b04f6ac1-maas-api-tls\") pod \"maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q\" (UID: \"10272086-df57-4b14-bca8-2aa5b04f6ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"
Sep 19 11:47:25.575205 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:25.575177    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10272086-df57-4b14-bca8-2aa5b04f6ac1-maas-api-tls\") pod \"maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q\" (UID: \"10272086-df57-4b14-bca8-2aa5b04f6ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"
Sep 19 11:47:25.761439 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:25.761351    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:25.837828 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:25.837789    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"
Sep 19 11:47:26.115390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:26.115363    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"]
Sep 19 11:47:26.117898 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:47:26.117857    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10272086_df57_4b14_bca8_2aa5b04f6ac1.slice/crio-a95073c4ffb26249f1a73e0539ec6ca16f2ac77c48dbd2dd3d8e72855e1df950 WatchSource:0}: Error finding container a95073c4ffb26249f1a73e0539ec6ca16f2ac77c48dbd2dd3d8e72855e1df950: Status 404 returned error can't find the container with id a95073c4ffb26249f1a73e0539ec6ca16f2ac77c48dbd2dd3d8e72855e1df950
Sep 19 11:47:26.283726 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:26.283689    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q" event={"ID":"10272086-df57-4b14-bca8-2aa5b04f6ac1","Type":"ContainerStarted","Data":"a95073c4ffb26249f1a73e0539ec6ca16f2ac77c48dbd2dd3d8e72855e1df950"}
Sep 19 11:47:26.533096 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:26.533067    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:26.632862 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:26.632828    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:27.291136 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:27.291092    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q" event={"ID":"10272086-df57-4b14-bca8-2aa5b04f6ac1","Type":"ContainerStarted","Data":"a3bbb7781ac4e55d46a102d0b417dfe07a5545851df09dc00c4b4e5ea562e1a9"}
Sep 19 11:47:27.291625 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:27.291174    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"
Sep 19 11:47:27.309148 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:27.309097    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q" podStartSLOduration=2.954715657 podStartE2EDuration="3.309077704s" totalImagesPullingTime="354.362047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:47:27.307596116 +0000 UTC m=+831.507625920" watchObservedRunningTime="2026-09-19 11:47:27.309077704 +0000 UTC m=+831.509107506"
Sep 19 11:47:27.714637 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:27.714577    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:27.775942 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:27.775894    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:27.960846 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:27.960806    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"]
Sep 19 11:47:27.961131 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:27.961103    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng" podUID="4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" containerName="istio-proxy" containerID="cri-o://69bdf708f919872c437c67f41392bc4b3e5a794c846bfc5c1936f62a182e9347" gracePeriod=30
Sep 19 11:47:27.976747 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:27.976659    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:32.283551 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:32.283516    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:32.343192 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:32.343153    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:32.422156 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:32.422119    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:32.522054 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:32.522018    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:33.223794 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.223719    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:47:33.253609 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.253578    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-token\") pod \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") "
Sep 19 11:47:33.253609 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.253611    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istiod-ca-cert\") pod \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") "
Sep 19 11:47:33.253885 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.253634    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5rcs\" (UniqueName: \"kubernetes.io/projected/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-kube-api-access-g5rcs\") pod \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") "
Sep 19 11:47:33.253885 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.253669    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-workload-certs\") pod \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") "
Sep 19 11:47:33.253885 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.253703    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-credential-socket\") pod \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") "
Sep 19 11:47:33.253885 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.253744    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-data\") pod \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") "
Sep 19 11:47:33.253885 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.253770    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-envoy\") pod \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") "
Sep 19 11:47:33.253885 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.253820    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-workload-socket\") pod \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") "
Sep 19 11:47:33.253885 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.253853    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-podinfo\") pod \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\" (UID: \"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088\") "
Sep 19 11:47:33.254274 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.254072    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" (UID: "4bf9df4b-c8e9-4455-9c2e-a7b6164ea088"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:47:33.254274 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.254214    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" (UID: "4bf9df4b-c8e9-4455-9c2e-a7b6164ea088"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:47:33.254274 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.254247    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-data" (OuterVolumeSpecName: "istio-data") pod "4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" (UID: "4bf9df4b-c8e9-4455-9c2e-a7b6164ea088"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:47:33.254433 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.254276    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" (UID: "4bf9df4b-c8e9-4455-9c2e-a7b6164ea088"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:47:33.254433 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.254299    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istiod-ca-cert\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:47:33.254701 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.254460    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" (UID: "4bf9df4b-c8e9-4455-9c2e-a7b6164ea088"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:47:33.256225 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.256192    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-token" (OuterVolumeSpecName: "istio-token") pod "4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" (UID: "4bf9df4b-c8e9-4455-9c2e-a7b6164ea088"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:47:33.256356 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.256323    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-kube-api-access-g5rcs" (OuterVolumeSpecName: "kube-api-access-g5rcs") pod "4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" (UID: "4bf9df4b-c8e9-4455-9c2e-a7b6164ea088"). InnerVolumeSpecName "kube-api-access-g5rcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:47:33.256420 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.256401    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" (UID: "4bf9df4b-c8e9-4455-9c2e-a7b6164ea088"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:47:33.256798 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.256781    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" (UID: "4bf9df4b-c8e9-4455-9c2e-a7b6164ea088"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 11:47:33.302188 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.302157    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"
Sep 19 11:47:33.326988 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.326952    2584 generic.go:356] "Generic (PLEG): container finished" podID="4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" containerID="69bdf708f919872c437c67f41392bc4b3e5a794c846bfc5c1936f62a182e9347" exitCode=0
Sep 19 11:47:33.327191 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.327047    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng" event={"ID":"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088","Type":"ContainerDied","Data":"69bdf708f919872c437c67f41392bc4b3e5a794c846bfc5c1936f62a182e9347"}
Sep 19 11:47:33.327637 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.327528    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng" event={"ID":"4bf9df4b-c8e9-4455-9c2e-a7b6164ea088","Type":"ContainerDied","Data":"ba2fa5960802a8507884e84aafd6b77ad49f18062f24f6cde36eacbea128fb14"}
Sep 19 11:47:33.327637 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.327577    2584 scope.go:117] "RemoveContainer" containerID="69bdf708f919872c437c67f41392bc4b3e5a794c846bfc5c1936f62a182e9347"
Sep 19 11:47:33.327803 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.327556    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"
Sep 19 11:47:33.351533 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.351506    2584 scope.go:117] "RemoveContainer" containerID="9578c3594ebf5961bb2ff5b975165155e314f25f3817200aefa5bcf5a2392ab7"
Sep 19 11:47:33.356242 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.356178    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-credential-socket\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:47:33.356242 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.356209    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-data\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:47:33.356242 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.356223    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-envoy\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:47:33.356242 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.356236    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-workload-socket\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:47:33.356242 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.356250    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-podinfo\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:47:33.356819 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.356263    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-istio-token\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:47:33.356819 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.356277    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5rcs\" (UniqueName: \"kubernetes.io/projected/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-kube-api-access-g5rcs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:47:33.356819 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.356291    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088-workload-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:47:33.360100 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.360018    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"]
Sep 19 11:47:33.363213 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.363182    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-37daed-openshift-default-7bc9b777cb-8k9ng"]
Sep 19 11:47:33.363898 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.363880    2584 scope.go:117] "RemoveContainer" containerID="69bdf708f919872c437c67f41392bc4b3e5a794c846bfc5c1936f62a182e9347"
Sep 19 11:47:33.364344 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:47:33.364321    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69bdf708f919872c437c67f41392bc4b3e5a794c846bfc5c1936f62a182e9347\": container with ID starting with 69bdf708f919872c437c67f41392bc4b3e5a794c846bfc5c1936f62a182e9347 not found: ID does not exist" containerID="69bdf708f919872c437c67f41392bc4b3e5a794c846bfc5c1936f62a182e9347"
Sep 19 11:47:33.364427 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.364358    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69bdf708f919872c437c67f41392bc4b3e5a794c846bfc5c1936f62a182e9347"} err="failed to get container status \"69bdf708f919872c437c67f41392bc4b3e5a794c846bfc5c1936f62a182e9347\": rpc error: code = NotFound desc = could not find container \"69bdf708f919872c437c67f41392bc4b3e5a794c846bfc5c1936f62a182e9347\": container with ID starting with 69bdf708f919872c437c67f41392bc4b3e5a794c846bfc5c1936f62a182e9347 not found: ID does not exist"
Sep 19 11:47:33.364427 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.364385    2584 scope.go:117] "RemoveContainer" containerID="9578c3594ebf5961bb2ff5b975165155e314f25f3817200aefa5bcf5a2392ab7"
Sep 19 11:47:33.365544 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:47:33.365518    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9578c3594ebf5961bb2ff5b975165155e314f25f3817200aefa5bcf5a2392ab7\": container with ID starting with 9578c3594ebf5961bb2ff5b975165155e314f25f3817200aefa5bcf5a2392ab7 not found: ID does not exist" containerID="9578c3594ebf5961bb2ff5b975165155e314f25f3817200aefa5bcf5a2392ab7"
Sep 19 11:47:33.365624 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.365553    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9578c3594ebf5961bb2ff5b975165155e314f25f3817200aefa5bcf5a2392ab7"} err="failed to get container status \"9578c3594ebf5961bb2ff5b975165155e314f25f3817200aefa5bcf5a2392ab7\": rpc error: code = NotFound desc = could not find container \"9578c3594ebf5961bb2ff5b975165155e314f25f3817200aefa5bcf5a2392ab7\": container with ID starting with 9578c3594ebf5961bb2ff5b975165155e314f25f3817200aefa5bcf5a2392ab7 not found: ID does not exist"
Sep 19 11:47:33.903971 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:33.903929    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" path="/var/lib/kubelet/pods/4bf9df4b-c8e9-4455-9c2e-a7b6164ea088/volumes"
Sep 19 11:47:42.979124 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:42.979086    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:43.756420 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:43.756380    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:46.947067 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:46.947029    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:47.852198 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:47.852164    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:48.135521 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:48.135430    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:48.500496 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:48.500409    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:49.273391 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:49.273350    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:50.057552 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:50.057519    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:50.150052 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:50.150019    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:50.247708 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:50.247674    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:50.343457 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:50.343430    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:50.961827 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:50.961792    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:51.055544 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:51.055509    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:51.116721 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:51.116646    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:52.019794 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:52.019741    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:52.615451 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:52.615413    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:52.741014 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:52.740978    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:58.478547 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:58.478503    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:58.580745 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:58.580705    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:47:59.143993 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:47:59.143951    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:04.901335 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:04.901299    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:04.964610 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:04.964571    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:05.137124 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:05.137094    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:05.445038 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:05.444987    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:05.544720 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:05.544686    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:07.224816 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.224779    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"]
Sep 19 11:48:07.225244 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.225184    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" containerName="istio-proxy"
Sep 19 11:48:07.225244 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.225196    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" containerName="istio-proxy"
Sep 19 11:48:07.225244 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.225208    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" containerName="istio-proxy"
Sep 19 11:48:07.225244 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.225213    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" containerName="istio-proxy"
Sep 19 11:48:07.225381 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.225303    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" containerName="istio-proxy"
Sep 19 11:48:07.233089 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.230004    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.233781 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.233761    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7eb15e\"/\"premium-w0-7eb15e-kserve-self-signed-certs\""
Sep 19 11:48:07.237678 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.237648    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"]
Sep 19 11:48:07.279025 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.278988    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-tmp-dir\") pod \"premium-w0-7eb15e-kserve-77b4884578-76g4l\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") " pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.279210 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.279034    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-model-cache\") pod \"premium-w0-7eb15e-kserve-77b4884578-76g4l\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") " pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.279210 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.279086    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-dshm\") pod \"premium-w0-7eb15e-kserve-77b4884578-76g4l\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") " pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.279210 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.279108    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0abba87-3d88-407c-abce-ba912965c5c5-tls-certs\") pod \"premium-w0-7eb15e-kserve-77b4884578-76g4l\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") " pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.279210 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.279187    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-home\") pod \"premium-w0-7eb15e-kserve-77b4884578-76g4l\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") " pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.279361 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.279232    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbnrv\" (UniqueName: \"kubernetes.io/projected/f0abba87-3d88-407c-abce-ba912965c5c5-kube-api-access-jbnrv\") pod \"premium-w0-7eb15e-kserve-77b4884578-76g4l\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") " pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.379738 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.379689    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-home\") pod \"premium-w0-7eb15e-kserve-77b4884578-76g4l\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") " pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.380011 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.379753    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbnrv\" (UniqueName: \"kubernetes.io/projected/f0abba87-3d88-407c-abce-ba912965c5c5-kube-api-access-jbnrv\") pod \"premium-w0-7eb15e-kserve-77b4884578-76g4l\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") " pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.380011 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.379802    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-tmp-dir\") pod \"premium-w0-7eb15e-kserve-77b4884578-76g4l\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") " pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.380011 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.379838    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-model-cache\") pod \"premium-w0-7eb15e-kserve-77b4884578-76g4l\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") " pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.380011 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.379881    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-dshm\") pod \"premium-w0-7eb15e-kserve-77b4884578-76g4l\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") " pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.380011 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.379948    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0abba87-3d88-407c-abce-ba912965c5c5-tls-certs\") pod \"premium-w0-7eb15e-kserve-77b4884578-76g4l\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") " pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.380286 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.380176    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-home\") pod \"premium-w0-7eb15e-kserve-77b4884578-76g4l\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") " pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.380286 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.380184    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-tmp-dir\") pod \"premium-w0-7eb15e-kserve-77b4884578-76g4l\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") " pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.380286 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.380257    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-model-cache\") pod \"premium-w0-7eb15e-kserve-77b4884578-76g4l\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") " pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.382406 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.382375    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-dshm\") pod \"premium-w0-7eb15e-kserve-77b4884578-76g4l\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") " pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.382565 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.382546    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0abba87-3d88-407c-abce-ba912965c5c5-tls-certs\") pod \"premium-w0-7eb15e-kserve-77b4884578-76g4l\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") " pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.388825 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.388797    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbnrv\" (UniqueName: \"kubernetes.io/projected/f0abba87-3d88-407c-abce-ba912965c5c5-kube-api-access-jbnrv\") pod \"premium-w0-7eb15e-kserve-77b4884578-76g4l\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") " pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.521945 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.521826    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:07.544131 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.544093    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:07.818531 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:07.818492    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"]
Sep 19 11:48:07.821102 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:48:07.821069    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0abba87_3d88_407c_abce_ba912965c5c5.slice/crio-bd1c504ebec125a3735a50ed118a5bb046124302aa79e03b34c2824e49107bea WatchSource:0}: Error finding container bd1c504ebec125a3735a50ed118a5bb046124302aa79e03b34c2824e49107bea: Status 404 returned error can't find the container with id bd1c504ebec125a3735a50ed118a5bb046124302aa79e03b34c2824e49107bea
Sep 19 11:48:08.516586 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:08.516546    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l" event={"ID":"f0abba87-3d88-407c-abce-ba912965c5c5","Type":"ContainerStarted","Data":"38db6cbc688f751f927f80f16a4b160cd68dc8ca06ee0eb8cf077065cd78ad1b"}
Sep 19 11:48:08.517035 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:08.516604    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l" event={"ID":"f0abba87-3d88-407c-abce-ba912965c5c5","Type":"ContainerStarted","Data":"bd1c504ebec125a3735a50ed118a5bb046124302aa79e03b34c2824e49107bea"}
Sep 19 11:48:08.539455 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:08.539401    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l" podStartSLOduration=1.539381905 podStartE2EDuration="1.539381905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:48:08.536958396 +0000 UTC m=+872.736988264" watchObservedRunningTime="2026-09-19 11:48:08.539381905 +0000 UTC m=+872.739411707"
Sep 19 11:48:08.544890 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:08.544862    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:10.418404 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:10.418373    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:11.086567 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:11.086469    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:11.178687 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:11.178656    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:11.357522 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:11.357424    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:12.160290 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:12.160258    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:12.945530 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:12.945496    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:13.665049 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:13.665011    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:14.274423 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.274385    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"]
Sep 19 11:48:14.274973 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.274927    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bf9df4b-c8e9-4455-9c2e-a7b6164ea088" containerName="istio-proxy"
Sep 19 11:48:14.278369 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.278344    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"
Sep 19 11:48:14.280863 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.280844    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c4de155d-serving-cert\""
Sep 19 11:48:14.288014 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.287987    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"]
Sep 19 11:48:14.302958 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.302929    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:14.348373 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.348335    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-openshift-service-ca\") pod \"maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq\" (UID: \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"
Sep 19 11:48:14.348553 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.348456    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-metrics-certs\") pod \"maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq\" (UID: \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"
Sep 19 11:48:14.348553 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.348520    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qbpb\" (UniqueName: \"kubernetes.io/projected/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-kube-api-access-7qbpb\") pod \"maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq\" (UID: \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"
Sep 19 11:48:14.348553 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.348551    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-maas-api-tls\") pod \"maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq\" (UID: \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"
Sep 19 11:48:14.404239 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.404201    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:14.449643 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.449603    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-maas-api-tls\") pod \"maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq\" (UID: \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"
Sep 19 11:48:14.449806 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.449667    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-openshift-service-ca\") pod \"maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq\" (UID: \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"
Sep 19 11:48:14.449806 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.449731    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-metrics-certs\") pod \"maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq\" (UID: \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"
Sep 19 11:48:14.449806 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:48:14.449754    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-serving-cert: secret "maas-api-e2e-mt-c4de155d-serving-cert" not found
Sep 19 11:48:14.449806 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.449771    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qbpb\" (UniqueName: \"kubernetes.io/projected/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-kube-api-access-7qbpb\") pod \"maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq\" (UID: \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"
Sep 19 11:48:14.450014 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:48:14.449814    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-maas-api-tls podName:d866f94d-201a-40c4-8e60-cf9b8ca8fba9 nodeName:}" failed. No retries permitted until 2026-09-19 11:48:14.949799182 +0000 UTC m=+879.149828967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-maas-api-tls") pod "maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq" (UID: "d866f94d-201a-40c4-8e60-cf9b8ca8fba9") : secret "maas-api-e2e-mt-c4de155d-serving-cert" not found
Sep 19 11:48:14.450374 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.450351    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-openshift-service-ca\") pod \"maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq\" (UID: \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"
Sep 19 11:48:14.452419 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.452395    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-metrics-certs\") pod \"maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq\" (UID: \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"
Sep 19 11:48:14.458267 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.458244    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qbpb\" (UniqueName: \"kubernetes.io/projected/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-kube-api-access-7qbpb\") pod \"maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq\" (UID: \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"
Sep 19 11:48:14.756184 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.756149    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:14.842196 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.842159    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:14.954397 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.954346    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-maas-api-tls\") pod \"maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq\" (UID: \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"
Sep 19 11:48:14.957131 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:14.957107    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-maas-api-tls\") pod \"maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq\" (UID: \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"
Sep 19 11:48:15.192504 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:15.192457    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"
Sep 19 11:48:15.469722 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:15.469695    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"]
Sep 19 11:48:15.471892 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:48:15.471863    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd866f94d_201a_40c4_8e60_cf9b8ca8fba9.slice/crio-32e28d4655e243c8e842db81a0cb247ca89fd5748376750949210062ff3849b9 WatchSource:0}: Error finding container 32e28d4655e243c8e842db81a0cb247ca89fd5748376750949210062ff3849b9: Status 404 returned error can't find the container with id 32e28d4655e243c8e842db81a0cb247ca89fd5748376750949210062ff3849b9
Sep 19 11:48:15.554251 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:15.554216    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq" event={"ID":"d866f94d-201a-40c4-8e60-cf9b8ca8fba9","Type":"ContainerStarted","Data":"32e28d4655e243c8e842db81a0cb247ca89fd5748376750949210062ff3849b9"}
Sep 19 11:48:16.560987 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:16.560938    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq" event={"ID":"d866f94d-201a-40c4-8e60-cf9b8ca8fba9","Type":"ContainerStarted","Data":"30f81433b37086005162d0051af993d4b57ba2b31fdcb9d468eed3c185ff5040"}
Sep 19 11:48:16.561501 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:16.561107    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"
Sep 19 11:48:16.580374 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:16.580314    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq" podStartSLOduration=2.000615789 podStartE2EDuration="2.580296934s" totalImagesPullingTime="579.681145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:48:16.577598159 +0000 UTC m=+880.777627965" watchObservedRunningTime="2026-09-19 11:48:16.580296934 +0000 UTC m=+880.780326738"
Sep 19 11:48:17.544562 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:17.544519    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:17.558635 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:17.558608    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:17.577871 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:17.577838    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:48:22.572663 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:22.572631    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"
Sep 19 11:48:24.920894 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:24.920860    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:25.020984 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:25.020949    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:36.033463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:36.033423    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:36.150029 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:36.149998    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:36.680453 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:36.680412    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:36.778775 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:36.778728    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:36.981136 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:36.981042    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:37.104039 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:37.104006    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:37.884335 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:37.884285    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:37.895145 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:37.895107    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/1.log"
Sep 19 11:48:37.896083 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:37.896062    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/1.log"
Sep 19 11:48:37.906314 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:37.906285    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6th98_2fed27e0-833f-49a8-b11e-8f61df6e6bd5/ovn-acl-logging/0.log"
Sep 19 11:48:37.906882 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:37.906863    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6th98_2fed27e0-833f-49a8-b11e-8f61df6e6bd5/ovn-acl-logging/0.log"
Sep 19 11:48:38.693445 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:38.693409    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:39.384576 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:39.384543    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:40.084565 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:40.084531    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:41.855582 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:41.855546    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:42.942079 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:42.942041    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:43.042698 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:43.042663    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:44.165807 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:44.165777    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:44.895566 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:44.895529    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:45.656700 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:45.656657    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:46.284831 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:46.284796    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:47.025757 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:47.025723    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:47.762011 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:47.761978    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:49.000201 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:49.000169    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:50.320376 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.320345    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"]
Sep 19 11:48:50.325397 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.325368    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.328343 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.328318    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-a527d4\"/\"openshift-service-ca.crt\""
Sep 19 11:48:50.328512 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.328354    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-a527d4\"/\"kube-root-ca.crt\""
Sep 19 11:48:50.329556 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.329537    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-a527d4\"/\"explicit-model-a527d4-kserve-self-signed-certs\""
Sep 19 11:48:50.329697 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.329678    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-a527d4\"/\"default-dockercfg-n2g8c\""
Sep 19 11:48:50.334081 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.334042    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"]
Sep 19 11:48:50.344590 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.344554    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:50.399769 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.399730    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-model-cache\") pod \"explicit-model-a527d4-kserve-7ddf6c56cb-s5x46\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") " pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.399769 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.399770    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-tmp-dir\") pod \"explicit-model-a527d4-kserve-7ddf6c56cb-s5x46\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") " pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.400028 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.399854    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfdwg\" (UniqueName: \"kubernetes.io/projected/38ba9341-3b77-4bdb-adca-49e55511e279-kube-api-access-mfdwg\") pod \"explicit-model-a527d4-kserve-7ddf6c56cb-s5x46\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") " pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.400028 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.399903    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38ba9341-3b77-4bdb-adca-49e55511e279-tls-certs\") pod \"explicit-model-a527d4-kserve-7ddf6c56cb-s5x46\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") " pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.400028 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.399974    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-home\") pod \"explicit-model-a527d4-kserve-7ddf6c56cb-s5x46\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") " pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.400028 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.400027    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-dshm\") pod \"explicit-model-a527d4-kserve-7ddf6c56cb-s5x46\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") " pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.500953 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.500900    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38ba9341-3b77-4bdb-adca-49e55511e279-tls-certs\") pod \"explicit-model-a527d4-kserve-7ddf6c56cb-s5x46\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") " pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.501155 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.500961    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-home\") pod \"explicit-model-a527d4-kserve-7ddf6c56cb-s5x46\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") " pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.501155 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.501020    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-dshm\") pod \"explicit-model-a527d4-kserve-7ddf6c56cb-s5x46\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") " pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.501155 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.501053    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-model-cache\") pod \"explicit-model-a527d4-kserve-7ddf6c56cb-s5x46\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") " pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.501155 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.501079    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-tmp-dir\") pod \"explicit-model-a527d4-kserve-7ddf6c56cb-s5x46\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") " pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.501155 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.501129    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfdwg\" (UniqueName: \"kubernetes.io/projected/38ba9341-3b77-4bdb-adca-49e55511e279-kube-api-access-mfdwg\") pod \"explicit-model-a527d4-kserve-7ddf6c56cb-s5x46\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") " pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.501467 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.501440    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-home\") pod \"explicit-model-a527d4-kserve-7ddf6c56cb-s5x46\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") " pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.501542 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.501464    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-model-cache\") pod \"explicit-model-a527d4-kserve-7ddf6c56cb-s5x46\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") " pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.501644 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.501619    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-tmp-dir\") pod \"explicit-model-a527d4-kserve-7ddf6c56cb-s5x46\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") " pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.504018 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.503995    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-dshm\") pod \"explicit-model-a527d4-kserve-7ddf6c56cb-s5x46\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") " pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.504130 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.504005    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38ba9341-3b77-4bdb-adca-49e55511e279-tls-certs\") pod \"explicit-model-a527d4-kserve-7ddf6c56cb-s5x46\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") " pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.511225 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.511200    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfdwg\" (UniqueName: \"kubernetes.io/projected/38ba9341-3b77-4bdb-adca-49e55511e279-kube-api-access-mfdwg\") pod \"explicit-model-a527d4-kserve-7ddf6c56cb-s5x46\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") " pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.637360 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.637269    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:50.912727 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:50.912702    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"]
Sep 19 11:48:50.914841 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:48:50.914809    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38ba9341_3b77_4bdb_adca_49e55511e279.slice/crio-6758804ee7f0708612d694bcc36dc234bcff7b427910b91b7a8a087ee59b4862 WatchSource:0}: Error finding container 6758804ee7f0708612d694bcc36dc234bcff7b427910b91b7a8a087ee59b4862: Status 404 returned error can't find the container with id 6758804ee7f0708612d694bcc36dc234bcff7b427910b91b7a8a087ee59b4862
Sep 19 11:48:51.763930 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:51.763869    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46" event={"ID":"38ba9341-3b77-4bdb-adca-49e55511e279","Type":"ContainerStarted","Data":"bfa393f71d0c52203a9a214e24a2420cf53e18f53e3ee4d65845442d4cf1b239"}
Sep 19 11:48:51.763930 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:51.763930    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46" event={"ID":"38ba9341-3b77-4bdb-adca-49e55511e279","Type":"ContainerStarted","Data":"6758804ee7f0708612d694bcc36dc234bcff7b427910b91b7a8a087ee59b4862"}
Sep 19 11:48:51.789807 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:51.789750    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46" podStartSLOduration=1.789731709 podStartE2EDuration="1.789731709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:48:51.787687034 +0000 UTC m=+915.987716851" watchObservedRunningTime="2026-09-19 11:48:51.789731709 +0000 UTC m=+915.989761512"
Sep 19 11:48:52.638174 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:52.638134    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:48:53.763240 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:53.763197    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"]
Sep 19 11:48:53.763748 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:53.763447    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq" podUID="d866f94d-201a-40c4-8e60-cf9b8ca8fba9" containerName="maas-api" containerID="cri-o://30f81433b37086005162d0051af993d4b57ba2b31fdcb9d468eed3c185ff5040" gracePeriod=30
Sep 19 11:48:53.791195 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:53.791165    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:53.857956 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:53.857902    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:53.920379 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:53.920348    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:53.990434 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:53.990386    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:54.018648 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.018583    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"
Sep 19 11:48:54.030024 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.029997    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-metrics-certs\") pod \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\" (UID: \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\") "
Sep 19 11:48:54.030172 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.030053    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-openshift-service-ca\") pod \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\" (UID: \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\") "
Sep 19 11:48:54.030172 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.030094    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-maas-api-tls\") pod \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\" (UID: \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\") "
Sep 19 11:48:54.030172 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.030123    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qbpb\" (UniqueName: \"kubernetes.io/projected/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-kube-api-access-7qbpb\") pod \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\" (UID: \"d866f94d-201a-40c4-8e60-cf9b8ca8fba9\") "
Sep 19 11:48:54.030506 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.030478    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d866f94d-201a-40c4-8e60-cf9b8ca8fba9" (UID: "d866f94d-201a-40c4-8e60-cf9b8ca8fba9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:48:54.032285 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.032255    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d866f94d-201a-40c4-8e60-cf9b8ca8fba9" (UID: "d866f94d-201a-40c4-8e60-cf9b8ca8fba9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:48:54.032389 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.032332    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-kube-api-access-7qbpb" (OuterVolumeSpecName: "kube-api-access-7qbpb") pod "d866f94d-201a-40c4-8e60-cf9b8ca8fba9" (UID: "d866f94d-201a-40c4-8e60-cf9b8ca8fba9"). InnerVolumeSpecName "kube-api-access-7qbpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:48:54.032389 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.032363    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d866f94d-201a-40c4-8e60-cf9b8ca8fba9" (UID: "d866f94d-201a-40c4-8e60-cf9b8ca8fba9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:48:54.064329 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.064282    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:54.130992 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.130961    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-openshift-service-ca\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:48:54.130992 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.130990    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-maas-api-tls\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:48:54.130992 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.131000    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qbpb\" (UniqueName: \"kubernetes.io/projected/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-kube-api-access-7qbpb\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:48:54.131234 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.131012    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d866f94d-201a-40c4-8e60-cf9b8ca8fba9-metrics-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:48:54.332209 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.332170    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:54.432806 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.432774    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:54.782904 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.782778    2584 generic.go:356] "Generic (PLEG): container finished" podID="d866f94d-201a-40c4-8e60-cf9b8ca8fba9" containerID="30f81433b37086005162d0051af993d4b57ba2b31fdcb9d468eed3c185ff5040" exitCode=0
Sep 19 11:48:54.782904 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.782891    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"
Sep 19 11:48:54.783408 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.782955    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq" event={"ID":"d866f94d-201a-40c4-8e60-cf9b8ca8fba9","Type":"ContainerDied","Data":"30f81433b37086005162d0051af993d4b57ba2b31fdcb9d468eed3c185ff5040"}
Sep 19 11:48:54.783408 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.783026    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq" event={"ID":"d866f94d-201a-40c4-8e60-cf9b8ca8fba9","Type":"ContainerDied","Data":"32e28d4655e243c8e842db81a0cb247ca89fd5748376750949210062ff3849b9"}
Sep 19 11:48:54.783408 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.783058    2584 scope.go:117] "RemoveContainer" containerID="30f81433b37086005162d0051af993d4b57ba2b31fdcb9d468eed3c185ff5040"
Sep 19 11:48:54.794852 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.794817    2584 scope.go:117] "RemoveContainer" containerID="30f81433b37086005162d0051af993d4b57ba2b31fdcb9d468eed3c185ff5040"
Sep 19 11:48:54.795215 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:48:54.795193    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f81433b37086005162d0051af993d4b57ba2b31fdcb9d468eed3c185ff5040\": container with ID starting with 30f81433b37086005162d0051af993d4b57ba2b31fdcb9d468eed3c185ff5040 not found: ID does not exist" containerID="30f81433b37086005162d0051af993d4b57ba2b31fdcb9d468eed3c185ff5040"
Sep 19 11:48:54.795304 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.795225    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f81433b37086005162d0051af993d4b57ba2b31fdcb9d468eed3c185ff5040"} err="failed to get container status \"30f81433b37086005162d0051af993d4b57ba2b31fdcb9d468eed3c185ff5040\": rpc error: code = NotFound desc = could not find container \"30f81433b37086005162d0051af993d4b57ba2b31fdcb9d468eed3c185ff5040\": container with ID starting with 30f81433b37086005162d0051af993d4b57ba2b31fdcb9d468eed3c185ff5040 not found: ID does not exist"
Sep 19 11:48:54.804628 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.804601    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"]
Sep 19 11:48:54.807887 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:54.807862    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4de155d-8d67b8b68-dpmgq"]
Sep 19 11:48:55.084658 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:55.084624    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:55.749775 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:55.749740    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:55.905297 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:55.905252    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d866f94d-201a-40c4-8e60-cf9b8ca8fba9" path="/var/lib/kubelet/pods/d866f94d-201a-40c4-8e60-cf9b8ca8fba9/volumes"
Sep 19 11:48:56.372818 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:56.372781    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:48:57.262631 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:48:57.262598    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:00.442463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:00.442438    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:00.498188 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:00.498158    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:00.638119 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:00.638080    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:49:00.651394 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:00.651366    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:49:00.816169 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:00.816139    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:00.828941 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:00.828888    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:49:01.049852 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:01.049789    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:01.149164 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:01.149045    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:01.269026 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:01.268994    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:06.986644 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:06.986609    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:07.094485 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:07.094428    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:07.186140 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:07.186106    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:07.321624 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:07.321586    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:07.694439 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:07.694335    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:09.496430 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:09.496377    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:12.358619 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:12.358584    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"]
Sep 19 11:49:12.382292 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:12.382261    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:12.806890 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:12.806798    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:13.571329 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:13.571300    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:13.969318 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:13.969221    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:14.072784 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:14.072744    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:14.269988 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:14.269879    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:14.853709 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:14.853657    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:15.149115 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:15.149018    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:15.249191 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:15.249152    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:15.493555 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:15.493463    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:15.986866 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:15.986831    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:16.748121 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:16.748082    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:17.554203 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:17.554172    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:17.774022 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:17.773989    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:17.873948 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:17.873883    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:18.466016 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:18.465972    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:19.286229 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:19.286193    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:19.953300 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:19.953270    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:20.757399 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:20.757364    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:21.479951 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.479898    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"]
Sep 19 11:49:21.480577 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.480554    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d866f94d-201a-40c4-8e60-cf9b8ca8fba9" containerName="maas-api"
Sep 19 11:49:21.480625 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.480584    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d866f94d-201a-40c4-8e60-cf9b8ca8fba9" containerName="maas-api"
Sep 19 11:49:21.480704 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.480692    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d866f94d-201a-40c4-8e60-cf9b8ca8fba9" containerName="maas-api"
Sep 19 11:49:21.486013 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.485986    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"
Sep 19 11:49:21.488523 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.488497    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-4bb32d-serving-cert\""
Sep 19 11:49:21.494384 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.494359    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"]
Sep 19 11:49:21.515413 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.515383    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:21.612349 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.612308    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf\" (UID: \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"
Sep 19 11:49:21.612537 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.612364    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdp85\" (UniqueName: \"kubernetes.io/projected/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-kube-api-access-sdp85\") pod \"maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf\" (UID: \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"
Sep 19 11:49:21.612537 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.612453    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf\" (UID: \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"
Sep 19 11:49:21.612537 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.612526    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf\" (UID: \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"
Sep 19 11:49:21.617134 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.617095    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:21.713362 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.713324    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf\" (UID: \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"
Sep 19 11:49:21.713551 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.713430    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf\" (UID: \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"
Sep 19 11:49:21.713551 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.713458    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdp85\" (UniqueName: \"kubernetes.io/projected/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-kube-api-access-sdp85\") pod \"maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf\" (UID: \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"
Sep 19 11:49:21.713551 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.713483    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf\" (UID: \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"
Sep 19 11:49:21.713551 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:49:21.713483    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-serving-cert: secret "maas-api-e2e-shared-a-w2-4bb32d-serving-cert" not found
Sep 19 11:49:21.713769 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:49:21.713625    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-maas-api-tls podName:2854ecea-0ccb-4ddc-ac6d-1f8a601d2789 nodeName:}" failed. No retries permitted until 2026-09-19 11:49:22.213605399 +0000 UTC m=+946.413635180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-maas-api-tls") pod "maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf" (UID: "2854ecea-0ccb-4ddc-ac6d-1f8a601d2789") : secret "maas-api-e2e-shared-a-w2-4bb32d-serving-cert" not found
Sep 19 11:49:21.714257 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.714232    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf\" (UID: \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"
Sep 19 11:49:21.716069 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.716046    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf\" (UID: \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"
Sep 19 11:49:21.721761 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:21.721739    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdp85\" (UniqueName: \"kubernetes.io/projected/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-kube-api-access-sdp85\") pod \"maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf\" (UID: \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"
Sep 19 11:49:22.065432 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:22.065382    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:22.163699 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:22.163660    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:22.217487 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:22.217448    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf\" (UID: \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"
Sep 19 11:49:22.220276 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:22.220246    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf\" (UID: \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"
Sep 19 11:49:22.236235 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:22.236198    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:22.400787 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:22.400692    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"
Sep 19 11:49:22.674542 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:22.674500    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"]
Sep 19 11:49:22.676673 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:49:22.676633    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2854ecea_0ccb_4ddc_ac6d_1f8a601d2789.slice/crio-aa47b49c6212816d5d8180affc3ab8d274a112c2b52a0d04a5a36b872bf92158 WatchSource:0}: Error finding container aa47b49c6212816d5d8180affc3ab8d274a112c2b52a0d04a5a36b872bf92158: Status 404 returned error can't find the container with id aa47b49c6212816d5d8180affc3ab8d274a112c2b52a0d04a5a36b872bf92158
Sep 19 11:49:22.678106 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:22.678089    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 11:49:22.877951 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:22.877898    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"]
Sep 19 11:49:22.878203 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:22.878164    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q" podUID="10272086-df57-4b14-bca8-2aa5b04f6ac1" containerName="maas-api" containerID="cri-o://a3bbb7781ac4e55d46a102d0b417dfe07a5545851df09dc00c4b4e5ea562e1a9" gracePeriod=30
Sep 19 11:49:22.908548 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:22.908518    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:22.939052 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:22.938951    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf" event={"ID":"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789","Type":"ContainerStarted","Data":"aa47b49c6212816d5d8180affc3ab8d274a112c2b52a0d04a5a36b872bf92158"}
Sep 19 11:49:23.007766 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.007722    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:23.100327 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.100294    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:23.148089 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.148060    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"
Sep 19 11:49:23.157541 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.157509    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:23.330549 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.330504    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5xnd\" (UniqueName: \"kubernetes.io/projected/10272086-df57-4b14-bca8-2aa5b04f6ac1-kube-api-access-m5xnd\") pod \"10272086-df57-4b14-bca8-2aa5b04f6ac1\" (UID: \"10272086-df57-4b14-bca8-2aa5b04f6ac1\") "
Sep 19 11:49:23.330719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.330560    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10272086-df57-4b14-bca8-2aa5b04f6ac1-maas-api-tls\") pod \"10272086-df57-4b14-bca8-2aa5b04f6ac1\" (UID: \"10272086-df57-4b14-bca8-2aa5b04f6ac1\") "
Sep 19 11:49:23.330719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.330595    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10272086-df57-4b14-bca8-2aa5b04f6ac1-metrics-certs\") pod \"10272086-df57-4b14-bca8-2aa5b04f6ac1\" (UID: \"10272086-df57-4b14-bca8-2aa5b04f6ac1\") "
Sep 19 11:49:23.330823 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.330764    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10272086-df57-4b14-bca8-2aa5b04f6ac1-openshift-service-ca\") pod \"10272086-df57-4b14-bca8-2aa5b04f6ac1\" (UID: \"10272086-df57-4b14-bca8-2aa5b04f6ac1\") "
Sep 19 11:49:23.331165 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.331124    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10272086-df57-4b14-bca8-2aa5b04f6ac1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "10272086-df57-4b14-bca8-2aa5b04f6ac1" (UID: "10272086-df57-4b14-bca8-2aa5b04f6ac1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:49:23.331473 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.331312    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10272086-df57-4b14-bca8-2aa5b04f6ac1-openshift-service-ca\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:49:23.332924 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.332885    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10272086-df57-4b14-bca8-2aa5b04f6ac1-kube-api-access-m5xnd" (OuterVolumeSpecName: "kube-api-access-m5xnd") pod "10272086-df57-4b14-bca8-2aa5b04f6ac1" (UID: "10272086-df57-4b14-bca8-2aa5b04f6ac1"). InnerVolumeSpecName "kube-api-access-m5xnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:49:23.333304 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.333278    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10272086-df57-4b14-bca8-2aa5b04f6ac1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "10272086-df57-4b14-bca8-2aa5b04f6ac1" (UID: "10272086-df57-4b14-bca8-2aa5b04f6ac1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:49:23.333304 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.333293    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10272086-df57-4b14-bca8-2aa5b04f6ac1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "10272086-df57-4b14-bca8-2aa5b04f6ac1" (UID: "10272086-df57-4b14-bca8-2aa5b04f6ac1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:49:23.432470 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.432427    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5xnd\" (UniqueName: \"kubernetes.io/projected/10272086-df57-4b14-bca8-2aa5b04f6ac1-kube-api-access-m5xnd\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:49:23.432470 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.432465    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10272086-df57-4b14-bca8-2aa5b04f6ac1-maas-api-tls\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:49:23.432470 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.432474    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10272086-df57-4b14-bca8-2aa5b04f6ac1-metrics-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:49:23.441548 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.441520    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:23.762468 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.762433    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:23.945232 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.945195    2584 generic.go:356] "Generic (PLEG): container finished" podID="10272086-df57-4b14-bca8-2aa5b04f6ac1" containerID="a3bbb7781ac4e55d46a102d0b417dfe07a5545851df09dc00c4b4e5ea562e1a9" exitCode=0
Sep 19 11:49:23.945421 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.945240    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"
Sep 19 11:49:23.945421 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.945270    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q" event={"ID":"10272086-df57-4b14-bca8-2aa5b04f6ac1","Type":"ContainerDied","Data":"a3bbb7781ac4e55d46a102d0b417dfe07a5545851df09dc00c4b4e5ea562e1a9"}
Sep 19 11:49:23.945421 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.945327    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q" event={"ID":"10272086-df57-4b14-bca8-2aa5b04f6ac1","Type":"ContainerDied","Data":"a95073c4ffb26249f1a73e0539ec6ca16f2ac77c48dbd2dd3d8e72855e1df950"}
Sep 19 11:49:23.945421 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.945358    2584 scope.go:117] "RemoveContainer" containerID="a3bbb7781ac4e55d46a102d0b417dfe07a5545851df09dc00c4b4e5ea562e1a9"
Sep 19 11:49:23.947325 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.947287    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf" event={"ID":"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789","Type":"ContainerStarted","Data":"1bd2156abf1920c73e3aec28d663c9f1a1e3f4fafa8230553d2589592714516b"}
Sep 19 11:49:23.947450 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.947358    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"
Sep 19 11:49:23.955110 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.955090    2584 scope.go:117] "RemoveContainer" containerID="a3bbb7781ac4e55d46a102d0b417dfe07a5545851df09dc00c4b4e5ea562e1a9"
Sep 19 11:49:23.955431 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:49:23.955410    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3bbb7781ac4e55d46a102d0b417dfe07a5545851df09dc00c4b4e5ea562e1a9\": container with ID starting with a3bbb7781ac4e55d46a102d0b417dfe07a5545851df09dc00c4b4e5ea562e1a9 not found: ID does not exist" containerID="a3bbb7781ac4e55d46a102d0b417dfe07a5545851df09dc00c4b4e5ea562e1a9"
Sep 19 11:49:23.955490 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.955443    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bbb7781ac4e55d46a102d0b417dfe07a5545851df09dc00c4b4e5ea562e1a9"} err="failed to get container status \"a3bbb7781ac4e55d46a102d0b417dfe07a5545851df09dc00c4b4e5ea562e1a9\": rpc error: code = NotFound desc = could not find container \"a3bbb7781ac4e55d46a102d0b417dfe07a5545851df09dc00c4b4e5ea562e1a9\": container with ID starting with a3bbb7781ac4e55d46a102d0b417dfe07a5545851df09dc00c4b4e5ea562e1a9 not found: ID does not exist"
Sep 19 11:49:23.966626 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.966565    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf" podStartSLOduration=2.623961573 podStartE2EDuration="2.966547761s" totalImagesPullingTime="342.586188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:49:23.965576826 +0000 UTC m=+948.165606630" watchObservedRunningTime="2026-09-19 11:49:23.966547761 +0000 UTC m=+948.166577601"
Sep 19 11:49:23.981465 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.981375    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"]
Sep 19 11:49:23.983369 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:23.983346    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a527d4-58cd6b899-qsj7q"]
Sep 19 11:49:24.656595 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:24.656558    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:25.453350 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:25.453315    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:25.903531 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:25.903491    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10272086-df57-4b14-bca8-2aa5b04f6ac1" path="/var/lib/kubelet/pods/10272086-df57-4b14-bca8-2aa5b04f6ac1/volumes"
Sep 19 11:49:26.129204 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:26.129168    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:26.748833 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:26.748791    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:27.380571 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.380531    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46" podUID="38ba9341-3b77-4bdb-adca-49e55511e279" containerName="main" containerID="cri-o://bfa393f71d0c52203a9a214e24a2420cf53e18f53e3ee4d65845442d4cf1b239" gracePeriod=45
Sep 19 11:49:27.648792 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.648720    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:49:27.771792 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.771750    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-model-cache\") pod \"38ba9341-3b77-4bdb-adca-49e55511e279\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") "
Sep 19 11:49:27.772008 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.771801    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfdwg\" (UniqueName: \"kubernetes.io/projected/38ba9341-3b77-4bdb-adca-49e55511e279-kube-api-access-mfdwg\") pod \"38ba9341-3b77-4bdb-adca-49e55511e279\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") "
Sep 19 11:49:27.772008 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.771827    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38ba9341-3b77-4bdb-adca-49e55511e279-tls-certs\") pod \"38ba9341-3b77-4bdb-adca-49e55511e279\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") "
Sep 19 11:49:27.772008 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.771858    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-dshm\") pod \"38ba9341-3b77-4bdb-adca-49e55511e279\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") "
Sep 19 11:49:27.772190 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.772037    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-home\") pod \"38ba9341-3b77-4bdb-adca-49e55511e279\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") "
Sep 19 11:49:27.772190 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.772064    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-tmp-dir\") pod \"38ba9341-3b77-4bdb-adca-49e55511e279\" (UID: \"38ba9341-3b77-4bdb-adca-49e55511e279\") "
Sep 19 11:49:27.772190 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.772104    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-model-cache" (OuterVolumeSpecName: "model-cache") pod "38ba9341-3b77-4bdb-adca-49e55511e279" (UID: "38ba9341-3b77-4bdb-adca-49e55511e279"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:49:27.772323 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.772295    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-home" (OuterVolumeSpecName: "home") pod "38ba9341-3b77-4bdb-adca-49e55511e279" (UID: "38ba9341-3b77-4bdb-adca-49e55511e279"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:49:27.772412 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.772376    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38ba9341-3b77-4bdb-adca-49e55511e279" (UID: "38ba9341-3b77-4bdb-adca-49e55511e279"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:49:27.772464 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.772452    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-model-cache\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:49:27.772497 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.772472    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-home\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:49:27.774269 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.774239    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38ba9341-3b77-4bdb-adca-49e55511e279-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38ba9341-3b77-4bdb-adca-49e55511e279" (UID: "38ba9341-3b77-4bdb-adca-49e55511e279"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:49:27.774372 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.774292    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ba9341-3b77-4bdb-adca-49e55511e279-kube-api-access-mfdwg" (OuterVolumeSpecName: "kube-api-access-mfdwg") pod "38ba9341-3b77-4bdb-adca-49e55511e279" (UID: "38ba9341-3b77-4bdb-adca-49e55511e279"). InnerVolumeSpecName "kube-api-access-mfdwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:49:27.774372 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.774313    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-dshm" (OuterVolumeSpecName: "dshm") pod "38ba9341-3b77-4bdb-adca-49e55511e279" (UID: "38ba9341-3b77-4bdb-adca-49e55511e279"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:49:27.873982 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.873905    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfdwg\" (UniqueName: \"kubernetes.io/projected/38ba9341-3b77-4bdb-adca-49e55511e279-kube-api-access-mfdwg\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:49:27.873982 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.873979    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38ba9341-3b77-4bdb-adca-49e55511e279-tls-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:49:27.873982 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.873990    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-dshm\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:49:27.874222 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.873997    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ba9341-3b77-4bdb-adca-49e55511e279-tmp-dir\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:49:27.970476 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.970443    2584 generic.go:356] "Generic (PLEG): container finished" podID="38ba9341-3b77-4bdb-adca-49e55511e279" containerID="bfa393f71d0c52203a9a214e24a2420cf53e18f53e3ee4d65845442d4cf1b239" exitCode=0
Sep 19 11:49:27.970681 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.970484    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"
Sep 19 11:49:27.970681 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.970533    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46" event={"ID":"38ba9341-3b77-4bdb-adca-49e55511e279","Type":"ContainerDied","Data":"bfa393f71d0c52203a9a214e24a2420cf53e18f53e3ee4d65845442d4cf1b239"}
Sep 19 11:49:27.970681 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.970571    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46" event={"ID":"38ba9341-3b77-4bdb-adca-49e55511e279","Type":"ContainerDied","Data":"6758804ee7f0708612d694bcc36dc234bcff7b427910b91b7a8a087ee59b4862"}
Sep 19 11:49:27.970681 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.970595    2584 scope.go:117] "RemoveContainer" containerID="bfa393f71d0c52203a9a214e24a2420cf53e18f53e3ee4d65845442d4cf1b239"
Sep 19 11:49:27.980042 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.980016    2584 scope.go:117] "RemoveContainer" containerID="bfa393f71d0c52203a9a214e24a2420cf53e18f53e3ee4d65845442d4cf1b239"
Sep 19 11:49:27.980394 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:49:27.980365    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa393f71d0c52203a9a214e24a2420cf53e18f53e3ee4d65845442d4cf1b239\": container with ID starting with bfa393f71d0c52203a9a214e24a2420cf53e18f53e3ee4d65845442d4cf1b239 not found: ID does not exist" containerID="bfa393f71d0c52203a9a214e24a2420cf53e18f53e3ee4d65845442d4cf1b239"
Sep 19 11:49:27.980467 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.980401    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa393f71d0c52203a9a214e24a2420cf53e18f53e3ee4d65845442d4cf1b239"} err="failed to get container status \"bfa393f71d0c52203a9a214e24a2420cf53e18f53e3ee4d65845442d4cf1b239\": rpc error: code = NotFound desc = could not find container \"bfa393f71d0c52203a9a214e24a2420cf53e18f53e3ee4d65845442d4cf1b239\": container with ID starting with bfa393f71d0c52203a9a214e24a2420cf53e18f53e3ee4d65845442d4cf1b239 not found: ID does not exist"
Sep 19 11:49:27.987454 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.987416    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"]
Sep 19 11:49:27.991188 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:27.991159    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-a527d4/explicit-model-a527d4-kserve-7ddf6c56cb-s5x46"]
Sep 19 11:49:29.038186 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:29.038153    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:29.092743 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:29.092705    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:29.300130 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:29.300009    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:29.753829 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:29.753790    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:29.903947 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:29.903896    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38ba9341-3b77-4bdb-adca-49e55511e279" path="/var/lib/kubelet/pods/38ba9341-3b77-4bdb-adca-49e55511e279/volumes"
Sep 19 11:49:29.959182 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:29.959154    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"
Sep 19 11:49:30.055571 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:30.055479    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:30.297597 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:30.297563    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:31.933506 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:31.933473    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:32.033806 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:32.033779    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:32.134657 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:32.134627    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:32.230692 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:32.230600    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:42.449754 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:42.449716    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:43.870849 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:43.870815    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:44.571105 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:44.571074    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:45.382690 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:45.382654    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:46.095570 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:46.095539    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:46.756379 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:46.756345    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:46.853522 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:46.853489    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:47.864122 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:47.864040    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:48.244344 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.244240    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:48.500414 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.500330    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"]
Sep 19 11:49:48.500794 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.500780    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ba9341-3b77-4bdb-adca-49e55511e279" containerName="main"
Sep 19 11:49:48.500850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.500796    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ba9341-3b77-4bdb-adca-49e55511e279" containerName="main"
Sep 19 11:49:48.500850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.500817    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10272086-df57-4b14-bca8-2aa5b04f6ac1" containerName="maas-api"
Sep 19 11:49:48.500850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.500822    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="10272086-df57-4b14-bca8-2aa5b04f6ac1" containerName="maas-api"
Sep 19 11:49:48.500964 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.500904    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="10272086-df57-4b14-bca8-2aa5b04f6ac1" containerName="maas-api"
Sep 19 11:49:48.500964 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.500935    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ba9341-3b77-4bdb-adca-49e55511e279" containerName="main"
Sep 19 11:49:48.505952 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.505930    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.508543 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.508517    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 19 11:49:48.508690 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.508547    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-176c86-openshift-default-dockercfg-bm2r5\""
Sep 19 11:49:48.514861 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.514837    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"]
Sep 19 11:49:48.526193 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.526160    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:48.576934 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.576889    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-podinfo\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.577104 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.577009    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-token\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.577104 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.577048    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-credential-socket\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.577104 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.577085    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-workload-socket\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.577213 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.577123    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmfjh\" (UniqueName: \"kubernetes.io/projected/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-kube-api-access-xmfjh\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.577213 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.577151    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-workload-certs\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.577213 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.577201    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-data\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.577331 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.577310    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istiod-ca-cert\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.577367 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.577356    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-envoy\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.678906 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.678863    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-data\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.679142 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.678962    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istiod-ca-cert\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.679142 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.678995    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-envoy\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.679142 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.679019    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-podinfo\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.679142 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.679048    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-token\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.679142 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.679081    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-credential-socket\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.679142 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.679116    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-workload-socket\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.679474 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.679158    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmfjh\" (UniqueName: \"kubernetes.io/projected/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-kube-api-access-xmfjh\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.679474 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.679196    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-workload-certs\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.679474 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.679359    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-data\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.679474 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.679458    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-workload-socket\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.679624 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.679495    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-credential-socket\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.679624 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.679575    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-workload-certs\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.679726 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.679707    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istiod-ca-cert\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.681489 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.681470    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-envoy\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.681752 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.681736    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-podinfo\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.687035 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.687001    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-token\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.687246 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.687230    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmfjh\" (UniqueName: \"kubernetes.io/projected/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-kube-api-access-xmfjh\") pod \"e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") " pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:48.819302 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:48.819268    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:49.036521 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:49.036489    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:49.097499 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:49.097474    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"]
Sep 19 11:49:49.099889 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:49:49.099850    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81bbf18e_1913_4789_a8b0_e8bf23d0b99a.slice/crio-429071f716b993627a4f367f7f79cde7190db2950ef835705af9eb8ee501df40 WatchSource:0}: Error finding container 429071f716b993627a4f367f7f79cde7190db2950ef835705af9eb8ee501df40: Status 404 returned error can't find the container with id 429071f716b993627a4f367f7f79cde7190db2950ef835705af9eb8ee501df40
Sep 19 11:49:49.102384 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:49.102340    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:49:49.102480 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:49.102427    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:49:49.102480 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:49.102463    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:49:50.095536 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:50.095497    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn" event={"ID":"81bbf18e-1913-4789-a8b0-e8bf23d0b99a","Type":"ContainerStarted","Data":"7db13862736d672e47f320be121ddd06ffa4d5cca24611981e08df799d827ace"}
Sep 19 11:49:50.095945 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:50.095563    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn" event={"ID":"81bbf18e-1913-4789-a8b0-e8bf23d0b99a","Type":"ContainerStarted","Data":"429071f716b993627a4f367f7f79cde7190db2950ef835705af9eb8ee501df40"}
Sep 19 11:49:50.115504 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:50.115447    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn" podStartSLOduration=2.115430101 podStartE2EDuration="2.115430101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:49:50.114494707 +0000 UTC m=+974.314524510" watchObservedRunningTime="2026-09-19 11:49:50.115430101 +0000 UTC m=+974.315459904"
Sep 19 11:49:50.819849 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:50.819810    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:50.825448 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:50.825423    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:51.100874 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:51.100775    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:51.101849 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:51.101829    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:49:51.330815 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:51.330786    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:51.961666 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:51.961613    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:52.061394 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:52.061356    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:52.161534 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:52.161491    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:52.863234 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:52.863188    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:53.594450 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:53.594412    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:54.367238 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:54.367200    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:55.301979 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:55.301940    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:55.403392 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:55.403340    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:55.888305 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:55.888274    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:55.989640 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:55.989584    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:49:56.057878 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:49:56.057848    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:18.647146 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:18.647111    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:19.431824 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:19.431789    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:19.940842 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:19.940806    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:20.146277 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:20.146244    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:21.156988 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:21.156936    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:21.786019 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:21.785991    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:22.108935 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:22.108882    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:22.435667 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:22.435579    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:22.860961 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:22.860906    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6444-55aee2"]
Sep 19 11:50:22.864613 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:22.864583    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6444-55aee2"
Sep 19 11:50:22.867421 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:22.867391    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qhqqh\""
Sep 19 11:50:22.868712 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:22.868689    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6444-55aee2"]
Sep 19 11:50:22.924740 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:22.924672    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kth78\" (UniqueName: \"kubernetes.io/projected/15352345-e7cd-48d7-b654-c3dcce32c6a2-kube-api-access-kth78\") pod \"test-curl-6444-55aee2\" (UID: \"15352345-e7cd-48d7-b654-c3dcce32c6a2\") " pod="opendatahub/test-curl-6444-55aee2"
Sep 19 11:50:23.026220 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:23.026184    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kth78\" (UniqueName: \"kubernetes.io/projected/15352345-e7cd-48d7-b654-c3dcce32c6a2-kube-api-access-kth78\") pod \"test-curl-6444-55aee2\" (UID: \"15352345-e7cd-48d7-b654-c3dcce32c6a2\") " pod="opendatahub/test-curl-6444-55aee2"
Sep 19 11:50:23.034473 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:23.034441    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kth78\" (UniqueName: \"kubernetes.io/projected/15352345-e7cd-48d7-b654-c3dcce32c6a2-kube-api-access-kth78\") pod \"test-curl-6444-55aee2\" (UID: \"15352345-e7cd-48d7-b654-c3dcce32c6a2\") " pod="opendatahub/test-curl-6444-55aee2"
Sep 19 11:50:23.176175 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:23.176086    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6444-55aee2"
Sep 19 11:50:23.470676 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:23.470651    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6444-55aee2"]
Sep 19 11:50:23.473054 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:50:23.473022    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15352345_e7cd_48d7_b654_c3dcce32c6a2.slice/crio-19d13b1a22711d845fcf722317572688532c3af1f5301adbc910b16d8c323216 WatchSource:0}: Error finding container 19d13b1a22711d845fcf722317572688532c3af1f5301adbc910b16d8c323216: Status 404 returned error can't find the container with id 19d13b1a22711d845fcf722317572688532c3af1f5301adbc910b16d8c323216
Sep 19 11:50:24.279061 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:24.279024    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6444-55aee2" event={"ID":"15352345-e7cd-48d7-b654-c3dcce32c6a2","Type":"ContainerStarted","Data":"b1620f93c67d5bccc2625ff783fe68a677801605526c78e7a5c735e318044345"}
Sep 19 11:50:24.279061 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:24.279077    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6444-55aee2" event={"ID":"15352345-e7cd-48d7-b654-c3dcce32c6a2","Type":"ContainerStarted","Data":"19d13b1a22711d845fcf722317572688532c3af1f5301adbc910b16d8c323216"}
Sep 19 11:50:24.293886 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:24.293813    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-6444-55aee2" podStartSLOduration=2.293791477 podStartE2EDuration="2.293791477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:50:24.292095259 +0000 UTC m=+1008.492125064" watchObservedRunningTime="2026-09-19 11:50:24.293791477 +0000 UTC m=+1008.493821281"
Sep 19 11:50:24.524462 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:24.524424    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"]
Sep 19 11:50:24.548840 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:24.548756    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:24.867805 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:24.867758    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6444-55aee2"]
Sep 19 11:50:24.869974 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:24.869942    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6444-55aee2"]
Sep 19 11:50:24.980931 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:24.980878    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6444-ea0327"]
Sep 19 11:50:24.981397 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:24.981382    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15352345-e7cd-48d7-b654-c3dcce32c6a2" containerName="test-curl-6444-55aee2"
Sep 19 11:50:24.981470 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:24.981399    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="15352345-e7cd-48d7-b654-c3dcce32c6a2" containerName="test-curl-6444-55aee2"
Sep 19 11:50:24.981503 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:24.981491    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="15352345-e7cd-48d7-b654-c3dcce32c6a2" containerName="test-curl-6444-55aee2"
Sep 19 11:50:24.984749 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:24.984727    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6444-ea0327"
Sep 19 11:50:25.011074 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:25.011003    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6444-ea0327"]
Sep 19 11:50:25.046071 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:25.046030    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-572bp\" (UniqueName: \"kubernetes.io/projected/e5286e50-65b5-4ca3-9c13-c89c574d28aa-kube-api-access-572bp\") pod \"test-curl-6444-ea0327\" (UID: \"e5286e50-65b5-4ca3-9c13-c89c574d28aa\") " pod="opendatahub/test-curl-6444-ea0327"
Sep 19 11:50:25.146881 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:25.146783    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-572bp\" (UniqueName: \"kubernetes.io/projected/e5286e50-65b5-4ca3-9c13-c89c574d28aa-kube-api-access-572bp\") pod \"test-curl-6444-ea0327\" (UID: \"e5286e50-65b5-4ca3-9c13-c89c574d28aa\") " pod="opendatahub/test-curl-6444-ea0327"
Sep 19 11:50:25.157257 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:25.157224    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-572bp\" (UniqueName: \"kubernetes.io/projected/e5286e50-65b5-4ca3-9c13-c89c574d28aa-kube-api-access-572bp\") pod \"test-curl-6444-ea0327\" (UID: \"e5286e50-65b5-4ca3-9c13-c89c574d28aa\") " pod="opendatahub/test-curl-6444-ea0327"
Sep 19 11:50:25.285543 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:50:25.285511    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-6444-55aee2" container="test-curl-6444-55aee2"
Sep 19 11:50:25.311075 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:25.311031    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6444-ea0327"
Sep 19 11:50:25.582588 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:25.582562    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6444-ea0327"]
Sep 19 11:50:25.584830 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:50:25.584799    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5286e50_65b5_4ca3_9c13_c89c574d28aa.slice/crio-f09085942a063e504280ad45adb6ffc1484e1fb264f661af1deb548be7fe4cfb WatchSource:0}: Error finding container f09085942a063e504280ad45adb6ffc1484e1fb264f661af1deb548be7fe4cfb: Status 404 returned error can't find the container with id f09085942a063e504280ad45adb6ffc1484e1fb264f661af1deb548be7fe4cfb
Sep 19 11:50:26.294812 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:26.294764    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-6444-55aee2" podUID="15352345-e7cd-48d7-b654-c3dcce32c6a2" containerName="test-curl-6444-55aee2" containerID="cri-o://b1620f93c67d5bccc2625ff783fe68a677801605526c78e7a5c735e318044345" gracePeriod=2
Sep 19 11:50:26.295667 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:26.295631    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6444-ea0327" event={"ID":"e5286e50-65b5-4ca3-9c13-c89c574d28aa","Type":"ContainerStarted","Data":"89d6fddf709c041330aaff9453567aa9de702e92a8d0d6cc893173dbf1b970b0"}
Sep 19 11:50:26.295799 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:26.295683    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6444-ea0327" event={"ID":"e5286e50-65b5-4ca3-9c13-c89c574d28aa","Type":"ContainerStarted","Data":"f09085942a063e504280ad45adb6ffc1484e1fb264f661af1deb548be7fe4cfb"}
Sep 19 11:50:26.313339 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:26.311596    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-6444-ea0327" podStartSLOduration=2.311570051 podStartE2EDuration="2.311570051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:50:26.308973962 +0000 UTC m=+1010.509003766" watchObservedRunningTime="2026-09-19 11:50:26.311570051 +0000 UTC m=+1010.511599855"
Sep 19 11:50:26.864245 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:26.864148    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6444-ea0327"]
Sep 19 11:50:26.868444 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:26.868407    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6444-ea0327"]
Sep 19 11:50:26.978136 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:26.978091    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6444-39491f"]
Sep 19 11:50:26.978602 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:26.978581    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5286e50-65b5-4ca3-9c13-c89c574d28aa" containerName="test-curl-6444-ea0327"
Sep 19 11:50:26.978687 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:26.978604    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5286e50-65b5-4ca3-9c13-c89c574d28aa" containerName="test-curl-6444-ea0327"
Sep 19 11:50:26.978756 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:26.978741    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5286e50-65b5-4ca3-9c13-c89c574d28aa" containerName="test-curl-6444-ea0327"
Sep 19 11:50:26.981987 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:26.981963    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6444-39491f"
Sep 19 11:50:26.987894 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:26.987861    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6444-39491f"]
Sep 19 11:50:27.067579 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:27.067536    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c54mx\" (UniqueName: \"kubernetes.io/projected/ce9dcc09-137d-448b-83ae-5dbf3a993947-kube-api-access-c54mx\") pod \"test-curl-6444-39491f\" (UID: \"ce9dcc09-137d-448b-83ae-5dbf3a993947\") " pod="opendatahub/test-curl-6444-39491f"
Sep 19 11:50:27.168740 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:27.168635    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c54mx\" (UniqueName: \"kubernetes.io/projected/ce9dcc09-137d-448b-83ae-5dbf3a993947-kube-api-access-c54mx\") pod \"test-curl-6444-39491f\" (UID: \"ce9dcc09-137d-448b-83ae-5dbf3a993947\") " pod="opendatahub/test-curl-6444-39491f"
Sep 19 11:50:27.176653 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:27.176618    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c54mx\" (UniqueName: \"kubernetes.io/projected/ce9dcc09-137d-448b-83ae-5dbf3a993947-kube-api-access-c54mx\") pod \"test-curl-6444-39491f\" (UID: \"ce9dcc09-137d-448b-83ae-5dbf3a993947\") " pod="opendatahub/test-curl-6444-39491f"
Sep 19 11:50:27.296365 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:27.296312    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6444-39491f"
Sep 19 11:50:27.301050 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:50:27.301013    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-6444-ea0327" container="test-curl-6444-ea0327"
Sep 19 11:50:27.578130 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:27.578094    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6444-39491f"]
Sep 19 11:50:27.581583 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:50:27.581551    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce9dcc09_137d_448b_83ae_5dbf3a993947.slice/crio-1d0652fa727e4d7704bba448da205d2a3ddfb9e616c10d9ed46ebf30df4c5b8e WatchSource:0}: Error finding container 1d0652fa727e4d7704bba448da205d2a3ddfb9e616c10d9ed46ebf30df4c5b8e: Status 404 returned error can't find the container with id 1d0652fa727e4d7704bba448da205d2a3ddfb9e616c10d9ed46ebf30df4c5b8e
Sep 19 11:50:28.310368 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:28.310329    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6444-39491f" event={"ID":"ce9dcc09-137d-448b-83ae-5dbf3a993947","Type":"ContainerStarted","Data":"fd54dab5007292e8a64ef06fcfc06bf85dafe9045c977a8b092d5f6689256aeb"}
Sep 19 11:50:28.310788 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:28.310380    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6444-39491f" event={"ID":"ce9dcc09-137d-448b-83ae-5dbf3a993947","Type":"ContainerStarted","Data":"1d0652fa727e4d7704bba448da205d2a3ddfb9e616c10d9ed46ebf30df4c5b8e"}
Sep 19 11:50:28.310788 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:28.310386    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-6444-ea0327" podUID="e5286e50-65b5-4ca3-9c13-c89c574d28aa" containerName="test-curl-6444-ea0327" containerID="cri-o://89d6fddf709c041330aaff9453567aa9de702e92a8d0d6cc893173dbf1b970b0" gracePeriod=2
Sep 19 11:50:28.324880 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:28.324817    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-6444-39491f" podStartSLOduration=2.324799086 podStartE2EDuration="2.324799086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:50:28.323211477 +0000 UTC m=+1012.523241280" watchObservedRunningTime="2026-09-19 11:50:28.324799086 +0000 UTC m=+1012.524828952"
Sep 19 11:50:28.507124 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:28.507089    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6444-55aee2"
Sep 19 11:50:28.581511 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:28.581475    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kth78\" (UniqueName: \"kubernetes.io/projected/15352345-e7cd-48d7-b654-c3dcce32c6a2-kube-api-access-kth78\") pod \"15352345-e7cd-48d7-b654-c3dcce32c6a2\" (UID: \"15352345-e7cd-48d7-b654-c3dcce32c6a2\") "
Sep 19 11:50:28.583712 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:28.583684    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15352345-e7cd-48d7-b654-c3dcce32c6a2-kube-api-access-kth78" (OuterVolumeSpecName: "kube-api-access-kth78") pod "15352345-e7cd-48d7-b654-c3dcce32c6a2" (UID: "15352345-e7cd-48d7-b654-c3dcce32c6a2"). InnerVolumeSpecName "kube-api-access-kth78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:50:28.682499 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:28.682474    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kth78\" (UniqueName: \"kubernetes.io/projected/15352345-e7cd-48d7-b654-c3dcce32c6a2-kube-api-access-kth78\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:50:28.874670 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:28.874585    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6444-39491f"]
Sep 19 11:50:28.880671 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:28.880640    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6444-39491f"]
Sep 19 11:50:28.987320 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:28.987280    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6444-7cbca9"]
Sep 19 11:50:28.987888 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:28.987871    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce9dcc09-137d-448b-83ae-5dbf3a993947" containerName="test-curl-6444-39491f"
Sep 19 11:50:28.987959 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:28.987893    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce9dcc09-137d-448b-83ae-5dbf3a993947" containerName="test-curl-6444-39491f"
Sep 19 11:50:28.988038 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:28.988026    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce9dcc09-137d-448b-83ae-5dbf3a993947" containerName="test-curl-6444-39491f"
Sep 19 11:50:28.991744 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:28.991708    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6444-7cbca9"
Sep 19 11:50:28.995253 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:28.995223    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6444-7cbca9"]
Sep 19 11:50:29.086646 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:29.086606    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7dxd\" (UniqueName: \"kubernetes.io/projected/7f6fbbb5-7ba5-4f65-828e-6c707e648c70-kube-api-access-j7dxd\") pod \"test-curl-6444-7cbca9\" (UID: \"7f6fbbb5-7ba5-4f65-828e-6c707e648c70\") " pod="opendatahub/test-curl-6444-7cbca9"
Sep 19 11:50:29.188219 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:29.188117    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7dxd\" (UniqueName: \"kubernetes.io/projected/7f6fbbb5-7ba5-4f65-828e-6c707e648c70-kube-api-access-j7dxd\") pod \"test-curl-6444-7cbca9\" (UID: \"7f6fbbb5-7ba5-4f65-828e-6c707e648c70\") " pod="opendatahub/test-curl-6444-7cbca9"
Sep 19 11:50:29.197160 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:29.197110    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7dxd\" (UniqueName: \"kubernetes.io/projected/7f6fbbb5-7ba5-4f65-828e-6c707e648c70-kube-api-access-j7dxd\") pod \"test-curl-6444-7cbca9\" (UID: \"7f6fbbb5-7ba5-4f65-828e-6c707e648c70\") " pod="opendatahub/test-curl-6444-7cbca9"
Sep 19 11:50:29.306044 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:29.305998    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6444-7cbca9"
Sep 19 11:50:29.322054 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:29.322023    2584 generic.go:356] "Generic (PLEG): container finished" podID="15352345-e7cd-48d7-b654-c3dcce32c6a2" containerID="b1620f93c67d5bccc2625ff783fe68a677801605526c78e7a5c735e318044345" exitCode=137
Sep 19 11:50:29.322440 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:29.322085    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6444-55aee2"
Sep 19 11:50:29.322440 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:29.322136    2584 scope.go:117] "RemoveContainer" containerID="b1620f93c67d5bccc2625ff783fe68a677801605526c78e7a5c735e318044345"
Sep 19 11:50:29.322559 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:50:29.322498    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-6444-39491f" container="test-curl-6444-39491f"
Sep 19 11:50:29.341167 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:29.341149    2584 scope.go:117] "RemoveContainer" containerID="b1620f93c67d5bccc2625ff783fe68a677801605526c78e7a5c735e318044345"
Sep 19 11:50:29.341460 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:50:29.341420    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6444-55aee2\" is forbidden: User \"system:node:ip-10-0-133-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-209.ec2.internal' and this object" podUID="15352345-e7cd-48d7-b654-c3dcce32c6a2" pod="opendatahub/test-curl-6444-55aee2"
Sep 19 11:50:29.341599 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:50:29.341574    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1620f93c67d5bccc2625ff783fe68a677801605526c78e7a5c735e318044345\": container with ID starting with b1620f93c67d5bccc2625ff783fe68a677801605526c78e7a5c735e318044345 not found: ID does not exist" containerID="b1620f93c67d5bccc2625ff783fe68a677801605526c78e7a5c735e318044345"
Sep 19 11:50:29.341686 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:29.341615    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1620f93c67d5bccc2625ff783fe68a677801605526c78e7a5c735e318044345"} err="failed to get container status \"b1620f93c67d5bccc2625ff783fe68a677801605526c78e7a5c735e318044345\": rpc error: code = NotFound desc = could not find container \"b1620f93c67d5bccc2625ff783fe68a677801605526c78e7a5c735e318044345\": container with ID starting with b1620f93c67d5bccc2625ff783fe68a677801605526c78e7a5c735e318044345 not found: ID does not exist"
Sep 19 11:50:29.583275 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:29.583245    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6444-7cbca9"]
Sep 19 11:50:29.585525 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:50:29.585492    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f6fbbb5_7ba5_4f65_828e_6c707e648c70.slice/crio-d019a27f6099fb43dfb466b39c36491efa0c4eaa732a3af55ada4bfbdb1b2796 WatchSource:0}: Error finding container d019a27f6099fb43dfb466b39c36491efa0c4eaa732a3af55ada4bfbdb1b2796: Status 404 returned error can't find the container with id d019a27f6099fb43dfb466b39c36491efa0c4eaa732a3af55ada4bfbdb1b2796
Sep 19 11:50:29.906365 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:29.906329    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15352345-e7cd-48d7-b654-c3dcce32c6a2" path="/var/lib/kubelet/pods/15352345-e7cd-48d7-b654-c3dcce32c6a2/volumes"
Sep 19 11:50:30.333475 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:30.333441    2584 generic.go:356] "Generic (PLEG): container finished" podID="81bbf18e-1913-4789-a8b0-e8bf23d0b99a" containerID="7db13862736d672e47f320be121ddd06ffa4d5cca24611981e08df799d827ace" exitCode=0
Sep 19 11:50:30.333952 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:30.333507    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn" event={"ID":"81bbf18e-1913-4789-a8b0-e8bf23d0b99a","Type":"ContainerDied","Data":"7db13862736d672e47f320be121ddd06ffa4d5cca24611981e08df799d827ace"}
Sep 19 11:50:30.334056 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:30.334024    2584 scope.go:117] "RemoveContainer" containerID="7db13862736d672e47f320be121ddd06ffa4d5cca24611981e08df799d827ace"
Sep 19 11:50:30.335797 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:30.335452    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:50:30.335797 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:30.335539    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:50:30.335797 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:30.335614    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:50:30.336075 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:30.335893    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6444-7cbca9" event={"ID":"7f6fbbb5-7ba5-4f65-828e-6c707e648c70","Type":"ContainerStarted","Data":"8686fc7e9de21c4323ebe707638b992e44a3a3bd21a5fde19e254a2c7841d870"}
Sep 19 11:50:30.336075 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:30.335969    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6444-7cbca9" event={"ID":"7f6fbbb5-7ba5-4f65-828e-6c707e648c70","Type":"ContainerStarted","Data":"d019a27f6099fb43dfb466b39c36491efa0c4eaa732a3af55ada4bfbdb1b2796"}
Sep 19 11:50:30.336075 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:30.335982    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-6444-39491f" podUID="ce9dcc09-137d-448b-83ae-5dbf3a993947" containerName="test-curl-6444-39491f" containerID="cri-o://fd54dab5007292e8a64ef06fcfc06bf85dafe9045c977a8b092d5f6689256aeb" gracePeriod=2
Sep 19 11:50:30.513130 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:30.513101    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6444-ea0327"
Sep 19 11:50:30.604836 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:30.604746    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-572bp\" (UniqueName: \"kubernetes.io/projected/e5286e50-65b5-4ca3-9c13-c89c574d28aa-kube-api-access-572bp\") pod \"e5286e50-65b5-4ca3-9c13-c89c574d28aa\" (UID: \"e5286e50-65b5-4ca3-9c13-c89c574d28aa\") "
Sep 19 11:50:30.607170 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:30.607140    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5286e50-65b5-4ca3-9c13-c89c574d28aa-kube-api-access-572bp" (OuterVolumeSpecName: "kube-api-access-572bp") pod "e5286e50-65b5-4ca3-9c13-c89c574d28aa" (UID: "e5286e50-65b5-4ca3-9c13-c89c574d28aa"). InnerVolumeSpecName "kube-api-access-572bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:50:30.705881 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:30.705844    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-572bp\" (UniqueName: \"kubernetes.io/projected/e5286e50-65b5-4ca3-9c13-c89c574d28aa-kube-api-access-572bp\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:50:30.930178 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:30.930054    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-6444-7cbca9" podStartSLOduration=2.9300354410000002 podStartE2EDuration="2.930035441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:50:30.377884205 +0000 UTC m=+1014.577914024" watchObservedRunningTime="2026-09-19 11:50:30.930035441 +0000 UTC m=+1015.130065244"
Sep 19 11:50:30.930798 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:30.930781    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6444-7cbca9"]
Sep 19 11:50:30.935321 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:30.935283    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6444-7cbca9"]
Sep 19 11:50:31.344203 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.344167    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn" event={"ID":"81bbf18e-1913-4789-a8b0-e8bf23d0b99a","Type":"ContainerStarted","Data":"a915d714ab2d957c45ec795bc46a53ed0cc46649d4805ca3bc2bf5fd38fdadd5"}
Sep 19 11:50:31.345513 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.345484    2584 generic.go:356] "Generic (PLEG): container finished" podID="e5286e50-65b5-4ca3-9c13-c89c574d28aa" containerID="89d6fddf709c041330aaff9453567aa9de702e92a8d0d6cc893173dbf1b970b0" exitCode=137
Sep 19 11:50:31.345657 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.345528    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6444-ea0327"
Sep 19 11:50:31.345657 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.345575    2584 scope.go:117] "RemoveContainer" containerID="89d6fddf709c041330aaff9453567aa9de702e92a8d0d6cc893173dbf1b970b0"
Sep 19 11:50:31.345984 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:50:31.345961    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-6444-7cbca9" container="test-curl-6444-7cbca9"
Sep 19 11:50:31.355837 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.355809    2584 scope.go:117] "RemoveContainer" containerID="89d6fddf709c041330aaff9453567aa9de702e92a8d0d6cc893173dbf1b970b0"
Sep 19 11:50:31.356273 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:50:31.356251    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d6fddf709c041330aaff9453567aa9de702e92a8d0d6cc893173dbf1b970b0\": container with ID starting with 89d6fddf709c041330aaff9453567aa9de702e92a8d0d6cc893173dbf1b970b0 not found: ID does not exist" containerID="89d6fddf709c041330aaff9453567aa9de702e92a8d0d6cc893173dbf1b970b0"
Sep 19 11:50:31.356364 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.356285    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d6fddf709c041330aaff9453567aa9de702e92a8d0d6cc893173dbf1b970b0"} err="failed to get container status \"89d6fddf709c041330aaff9453567aa9de702e92a8d0d6cc893173dbf1b970b0\": rpc error: code = NotFound desc = could not find container \"89d6fddf709c041330aaff9453567aa9de702e92a8d0d6cc893173dbf1b970b0\": container with ID starting with 89d6fddf709c041330aaff9453567aa9de702e92a8d0d6cc893173dbf1b970b0 not found: ID does not exist"
Sep 19 11:50:31.367708 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:50:31.367669    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6444-ea0327\" is forbidden: User \"system:node:ip-10-0-133-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-209.ec2.internal' and this object" podUID="e5286e50-65b5-4ca3-9c13-c89c574d28aa" pod="opendatahub/test-curl-6444-ea0327"
Sep 19 11:50:31.504639 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.504601    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:31.771689 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.771596    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"]
Sep 19 11:50:31.772103 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.772083    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f6fbbb5-7ba5-4f65-828e-6c707e648c70" containerName="test-curl-6444-7cbca9"
Sep 19 11:50:31.772103 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.772102    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f6fbbb5-7ba5-4f65-828e-6c707e648c70" containerName="test-curl-6444-7cbca9"
Sep 19 11:50:31.772266 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.772172    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f6fbbb5-7ba5-4f65-828e-6c707e648c70" containerName="test-curl-6444-7cbca9"
Sep 19 11:50:31.774382 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.774363    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:31.776927 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.776892    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fcfc8fcc-openshift-default-dockercfg-wjttz\""
Sep 19 11:50:31.786011 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.785983    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"]
Sep 19 11:50:31.793282 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.793248    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:31.799460 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:50:31.799418    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6444-ea0327\" is forbidden: User \"system:node:ip-10-0-133-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-209.ec2.internal' and this object" podUID="e5286e50-65b5-4ca3-9c13-c89c574d28aa" pod="opendatahub/test-curl-6444-ea0327"
Sep 19 11:50:31.819454 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.819420    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:50:31.820677 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.820649    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 19 11:50:31.820818 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.820723    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn" podUID="81bbf18e-1913-4789-a8b0-e8bf23d0b99a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 19 11:50:31.858820 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.858774    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:31.913759 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.913711    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5286e50-65b5-4ca3-9c13-c89c574d28aa" path="/var/lib/kubelet/pods/e5286e50-65b5-4ca3-9c13-c89c574d28aa/volumes"
Sep 19 11:50:31.917951 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.917889    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:31.919818 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.919792    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138969f5-591e-4ede-8d44-0f568c959000-istio-podinfo\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:31.919894 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.919847    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-istio-data\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:31.919894 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.919867    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138969f5-591e-4ede-8d44-0f568c959000-istio-token\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:31.920016 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.919930    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-istio-envoy\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:31.920016 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.919999    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-workload-socket\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:31.920123 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.920064    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-credential-socket\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:31.920123 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.920099    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138969f5-591e-4ede-8d44-0f568c959000-istiod-ca-cert\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:31.920208 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.920125    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lv6t\" (UniqueName: \"kubernetes.io/projected/138969f5-591e-4ede-8d44-0f568c959000-kube-api-access-7lv6t\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:31.920208 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:31.920178    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-workload-certs\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.021605 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.021562    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-istio-envoy\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.021605 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.021609    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-workload-socket\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.021966 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.021655    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-credential-socket\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.021966 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.021684    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138969f5-591e-4ede-8d44-0f568c959000-istiod-ca-cert\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.021966 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.021712    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lv6t\" (UniqueName: \"kubernetes.io/projected/138969f5-591e-4ede-8d44-0f568c959000-kube-api-access-7lv6t\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.021966 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.021746    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-workload-certs\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.022273 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.022082    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138969f5-591e-4ede-8d44-0f568c959000-istio-podinfo\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.022273 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.022153    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-workload-socket\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.022273 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.022186    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-workload-certs\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.022273 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.022214    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-istio-data\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.022273 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.022250    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138969f5-591e-4ede-8d44-0f568c959000-istio-token\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.022530 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.022437    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-credential-socket\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.022648 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.022621    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138969f5-591e-4ede-8d44-0f568c959000-istiod-ca-cert\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.022648 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.022642    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-istio-data\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.024744 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.024716    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-istio-envoy\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.024842 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.024807    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138969f5-591e-4ede-8d44-0f568c959000-istio-podinfo\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.030246 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.030213    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138969f5-591e-4ede-8d44-0f568c959000-istio-token\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.030431 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.030406    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lv6t\" (UniqueName: \"kubernetes.io/projected/138969f5-591e-4ede-8d44-0f568c959000-kube-api-access-7lv6t\") pod \"e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") " pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.085289 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.085247    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:32.352661 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.352598    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-6444-7cbca9" podUID="7f6fbbb5-7ba5-4f65-828e-6c707e648c70" containerName="test-curl-6444-7cbca9" containerID="cri-o://8686fc7e9de21c4323ebe707638b992e44a3a3bd21a5fde19e254a2c7841d870" gracePeriod=2
Sep 19 11:50:32.385261 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.385125    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"]
Sep 19 11:50:32.399201 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.398340    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:50:32.399201 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.398437    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:50:32.399201 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.398479    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:50:32.561797 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.561768    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6444-39491f"
Sep 19 11:50:32.629088 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.628998    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c54mx\" (UniqueName: \"kubernetes.io/projected/ce9dcc09-137d-448b-83ae-5dbf3a993947-kube-api-access-c54mx\") pod \"ce9dcc09-137d-448b-83ae-5dbf3a993947\" (UID: \"ce9dcc09-137d-448b-83ae-5dbf3a993947\") "
Sep 19 11:50:32.631822 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.631790    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9dcc09-137d-448b-83ae-5dbf3a993947-kube-api-access-c54mx" (OuterVolumeSpecName: "kube-api-access-c54mx") pod "ce9dcc09-137d-448b-83ae-5dbf3a993947" (UID: "ce9dcc09-137d-448b-83ae-5dbf3a993947"). InnerVolumeSpecName "kube-api-access-c54mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:50:32.730516 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.730472    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c54mx\" (UniqueName: \"kubernetes.io/projected/ce9dcc09-137d-448b-83ae-5dbf3a993947-kube-api-access-c54mx\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:50:32.820266 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.820222    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 19 11:50:32.820485 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:32.820299    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn" podUID="81bbf18e-1913-4789-a8b0-e8bf23d0b99a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 19 11:50:33.359809 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:33.359768    2584 generic.go:356] "Generic (PLEG): container finished" podID="ce9dcc09-137d-448b-83ae-5dbf3a993947" containerID="fd54dab5007292e8a64ef06fcfc06bf85dafe9045c977a8b092d5f6689256aeb" exitCode=137
Sep 19 11:50:33.360308 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:33.359815    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6444-39491f"
Sep 19 11:50:33.360308 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:33.359873    2584 scope.go:117] "RemoveContainer" containerID="fd54dab5007292e8a64ef06fcfc06bf85dafe9045c977a8b092d5f6689256aeb"
Sep 19 11:50:33.361967 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:33.361937    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b" event={"ID":"138969f5-591e-4ede-8d44-0f568c959000","Type":"ContainerStarted","Data":"744ccd96d9e1198bdb5341c19cc9f6ab05149acf9bf47c125303a76055d4d5a1"}
Sep 19 11:50:33.362072 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:33.361999    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b" event={"ID":"138969f5-591e-4ede-8d44-0f568c959000","Type":"ContainerStarted","Data":"b1b96b900c3d88cec2693d79454db76ce3465f0d8dcaef5a9743663131b20ab3"}
Sep 19 11:50:33.370160 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:33.370137    2584 scope.go:117] "RemoveContainer" containerID="fd54dab5007292e8a64ef06fcfc06bf85dafe9045c977a8b092d5f6689256aeb"
Sep 19 11:50:33.370497 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:50:33.370479    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd54dab5007292e8a64ef06fcfc06bf85dafe9045c977a8b092d5f6689256aeb\": container with ID starting with fd54dab5007292e8a64ef06fcfc06bf85dafe9045c977a8b092d5f6689256aeb not found: ID does not exist" containerID="fd54dab5007292e8a64ef06fcfc06bf85dafe9045c977a8b092d5f6689256aeb"
Sep 19 11:50:33.370550 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:33.370506    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd54dab5007292e8a64ef06fcfc06bf85dafe9045c977a8b092d5f6689256aeb"} err="failed to get container status \"fd54dab5007292e8a64ef06fcfc06bf85dafe9045c977a8b092d5f6689256aeb\": rpc error: code = NotFound desc = could not find container \"fd54dab5007292e8a64ef06fcfc06bf85dafe9045c977a8b092d5f6689256aeb\": container with ID starting with fd54dab5007292e8a64ef06fcfc06bf85dafe9045c977a8b092d5f6689256aeb not found: ID does not exist"
Sep 19 11:50:33.382507 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:50:33.382470    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6444-39491f\" is forbidden: User \"system:node:ip-10-0-133-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-209.ec2.internal' and this object" podUID="ce9dcc09-137d-448b-83ae-5dbf3a993947" pod="opendatahub/test-curl-6444-39491f"
Sep 19 11:50:33.383198 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:33.383158    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b" podStartSLOduration=2.38314159 podStartE2EDuration="2.38314159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:50:33.380264235 +0000 UTC m=+1017.580294038" watchObservedRunningTime="2026-09-19 11:50:33.38314159 +0000 UTC m=+1017.583171393"
Sep 19 11:50:33.820545 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:33.820499    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 19 11:50:33.820745 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:33.820558    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn" podUID="81bbf18e-1913-4789-a8b0-e8bf23d0b99a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 19 11:50:33.908871 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:33.908819    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce9dcc09-137d-448b-83ae-5dbf3a993947" path="/var/lib/kubelet/pods/ce9dcc09-137d-448b-83ae-5dbf3a993947/volumes"
Sep 19 11:50:34.086305 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:34.086219    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:34.105517 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:34.105477    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:34.369699 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:34.369468    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:34.370354 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:34.370335    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:50:34.442578 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:34.442536    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:34.623531 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:34.623306    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6444-7cbca9"
Sep 19 11:50:34.749859 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:34.749829    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7dxd\" (UniqueName: \"kubernetes.io/projected/7f6fbbb5-7ba5-4f65-828e-6c707e648c70-kube-api-access-j7dxd\") pod \"7f6fbbb5-7ba5-4f65-828e-6c707e648c70\" (UID: \"7f6fbbb5-7ba5-4f65-828e-6c707e648c70\") "
Sep 19 11:50:34.752173 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:34.752144    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f6fbbb5-7ba5-4f65-828e-6c707e648c70-kube-api-access-j7dxd" (OuterVolumeSpecName: "kube-api-access-j7dxd") pod "7f6fbbb5-7ba5-4f65-828e-6c707e648c70" (UID: "7f6fbbb5-7ba5-4f65-828e-6c707e648c70"). InnerVolumeSpecName "kube-api-access-j7dxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:50:34.820494 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:34.820454    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 19 11:50:34.820675 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:34.820514    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn" podUID="81bbf18e-1913-4789-a8b0-e8bf23d0b99a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 19 11:50:34.851192 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:34.851158    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7dxd\" (UniqueName: \"kubernetes.io/projected/7f6fbbb5-7ba5-4f65-828e-6c707e648c70-kube-api-access-j7dxd\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:50:35.146691 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:35.146646    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:35.252944 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:35.252888    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:35.346975 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:35.346922    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:35.377555 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:35.377505    2584 generic.go:356] "Generic (PLEG): container finished" podID="7f6fbbb5-7ba5-4f65-828e-6c707e648c70" containerID="8686fc7e9de21c4323ebe707638b992e44a3a3bd21a5fde19e254a2c7841d870" exitCode=137
Sep 19 11:50:35.378123 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:35.377636    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6444-7cbca9"
Sep 19 11:50:35.378123 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:35.377725    2584 scope.go:117] "RemoveContainer" containerID="8686fc7e9de21c4323ebe707638b992e44a3a3bd21a5fde19e254a2c7841d870"
Sep 19 11:50:35.387188 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:35.387007    2584 scope.go:117] "RemoveContainer" containerID="8686fc7e9de21c4323ebe707638b992e44a3a3bd21a5fde19e254a2c7841d870"
Sep 19 11:50:35.387455 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:50:35.387423    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8686fc7e9de21c4323ebe707638b992e44a3a3bd21a5fde19e254a2c7841d870\": container with ID starting with 8686fc7e9de21c4323ebe707638b992e44a3a3bd21a5fde19e254a2c7841d870 not found: ID does not exist" containerID="8686fc7e9de21c4323ebe707638b992e44a3a3bd21a5fde19e254a2c7841d870"
Sep 19 11:50:35.387587 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:35.387501    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8686fc7e9de21c4323ebe707638b992e44a3a3bd21a5fde19e254a2c7841d870"} err="failed to get container status \"8686fc7e9de21c4323ebe707638b992e44a3a3bd21a5fde19e254a2c7841d870\": rpc error: code = NotFound desc = could not find container \"8686fc7e9de21c4323ebe707638b992e44a3a3bd21a5fde19e254a2c7841d870\": container with ID starting with 8686fc7e9de21c4323ebe707638b992e44a3a3bd21a5fde19e254a2c7841d870 not found: ID does not exist"
Sep 19 11:50:35.395257 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:50:35.395207    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6444-7cbca9\" is forbidden: User \"system:node:ip-10-0-133-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-209.ec2.internal' and this object" podUID="7f6fbbb5-7ba5-4f65-828e-6c707e648c70" pod="opendatahub/test-curl-6444-7cbca9"
Sep 19 11:50:35.820544 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:35.820502    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 19 11:50:35.820743 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:35.820565    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn" podUID="81bbf18e-1913-4789-a8b0-e8bf23d0b99a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 19 11:50:35.909237 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:35.909192    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f6fbbb5-7ba5-4f65-828e-6c707e648c70" path="/var/lib/kubelet/pods/7f6fbbb5-7ba5-4f65-828e-6c707e648c70/volumes"
Sep 19 11:50:36.056501 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:36.056465    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:36.101706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:36.101570    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:50:36.719997 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:36.719959    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:36.819985 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:36.819944    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 19 11:50:36.820172 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:36.820008    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn" podUID="81bbf18e-1913-4789-a8b0-e8bf23d0b99a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 19 11:50:37.380193 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:37.380161    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:37.824739 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:37.824706    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:50:37.825768 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:37.825744    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:50:38.162872 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:38.162777    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:38.771379 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:38.771337    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"]
Sep 19 11:50:38.774873 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:38.774850    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"
Sep 19 11:50:38.777574 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:38.777545    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fcfc8fcc-serving-cert\""
Sep 19 11:50:38.784897 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:38.784858    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"]
Sep 19 11:50:38.830067 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:38.830034    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:38.896514 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:38.896463    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5k9d\" (UniqueName: \"kubernetes.io/projected/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-kube-api-access-m5k9d\") pod \"maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4\" (UID: \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"
Sep 19 11:50:38.896706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:38.896601    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-metrics-certs\") pod \"maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4\" (UID: \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"
Sep 19 11:50:38.896706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:38.896628    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-maas-api-tls\") pod \"maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4\" (UID: \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"
Sep 19 11:50:38.896706 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:38.896676    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-openshift-service-ca\") pod \"maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4\" (UID: \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"
Sep 19 11:50:38.930486 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:38.930452    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:38.997929 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:38.997883    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5k9d\" (UniqueName: \"kubernetes.io/projected/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-kube-api-access-m5k9d\") pod \"maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4\" (UID: \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"
Sep 19 11:50:38.998135 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:38.998089    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-metrics-certs\") pod \"maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4\" (UID: \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"
Sep 19 11:50:38.998135 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:38.998120    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-maas-api-tls\") pod \"maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4\" (UID: \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"
Sep 19 11:50:38.998259 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:38.998161    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-openshift-service-ca\") pod \"maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4\" (UID: \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"
Sep 19 11:50:38.998344 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:50:38.998315    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-serving-cert: secret "maas-api-e2e-mt-fcfc8fcc-serving-cert" not found
Sep 19 11:50:38.998423 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:50:38.998410    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-maas-api-tls podName:f38ac71e-49f6-44fb-a3c7-8417fa6c36fe nodeName:}" failed. No retries permitted until 2026-09-19 11:50:39.498386792 +0000 UTC m=+1023.698416591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-maas-api-tls") pod "maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4" (UID: "f38ac71e-49f6-44fb-a3c7-8417fa6c36fe") : secret "maas-api-e2e-mt-fcfc8fcc-serving-cert" not found
Sep 19 11:50:38.998949 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:38.998899    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-openshift-service-ca\") pod \"maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4\" (UID: \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"
Sep 19 11:50:39.000808 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.000786    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-metrics-certs\") pod \"maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4\" (UID: \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"
Sep 19 11:50:39.006853 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.006819    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5k9d\" (UniqueName: \"kubernetes.io/projected/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-kube-api-access-m5k9d\") pod \"maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4\" (UID: \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"
Sep 19 11:50:39.266387 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.266354    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:39.393262 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.393224    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:39.452530 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.452500    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:39.502850 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.502818    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-maas-api-tls\") pod \"maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4\" (UID: \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"
Sep 19 11:50:39.505510 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.505482    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-maas-api-tls\") pod \"maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4\" (UID: \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"
Sep 19 11:50:39.555676 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.555583    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht" podUID="4c151ef3-0db2-4693-bb09-1ba0e66c3968" containerName="main" containerID="cri-o://bbe782c7f108ff72f6d7c76035ba49b084bd63b1afd6042b4aa34cfa288bfe17" gracePeriod=45
Sep 19 11:50:39.687808 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.687474    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"
Sep 19 11:50:39.819801 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.819773    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:50:39.907456 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.907425    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-home\") pod \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") "
Sep 19 11:50:39.907891 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.907467    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-dshm\") pod \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") "
Sep 19 11:50:39.907891 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.907504    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncm6z\" (UniqueName: \"kubernetes.io/projected/4c151ef3-0db2-4693-bb09-1ba0e66c3968-kube-api-access-ncm6z\") pod \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") "
Sep 19 11:50:39.907891 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.907541    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-model-cache\") pod \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") "
Sep 19 11:50:39.907891 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.907565    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-tmp-dir\") pod \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") "
Sep 19 11:50:39.907891 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.907682    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-home" (OuterVolumeSpecName: "home") pod "4c151ef3-0db2-4693-bb09-1ba0e66c3968" (UID: "4c151ef3-0db2-4693-bb09-1ba0e66c3968"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:50:39.907891 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.907687    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c151ef3-0db2-4693-bb09-1ba0e66c3968-tls-certs\") pod \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\" (UID: \"4c151ef3-0db2-4693-bb09-1ba0e66c3968\") "
Sep 19 11:50:39.907891 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.907794    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c151ef3-0db2-4693-bb09-1ba0e66c3968" (UID: "4c151ef3-0db2-4693-bb09-1ba0e66c3968"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:50:39.907891 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.907833    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c151ef3-0db2-4693-bb09-1ba0e66c3968" (UID: "4c151ef3-0db2-4693-bb09-1ba0e66c3968"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:50:39.908272 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.908018    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-home\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:50:39.908272 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.908050    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-model-cache\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:50:39.908272 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.908067    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-tmp-dir\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:50:39.909956 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.909935    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-dshm" (OuterVolumeSpecName: "dshm") pod "4c151ef3-0db2-4693-bb09-1ba0e66c3968" (UID: "4c151ef3-0db2-4693-bb09-1ba0e66c3968"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:50:39.910033 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.910001    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c151ef3-0db2-4693-bb09-1ba0e66c3968-kube-api-access-ncm6z" (OuterVolumeSpecName: "kube-api-access-ncm6z") pod "4c151ef3-0db2-4693-bb09-1ba0e66c3968" (UID: "4c151ef3-0db2-4693-bb09-1ba0e66c3968"). InnerVolumeSpecName "kube-api-access-ncm6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:50:39.910033 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:39.909997    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c151ef3-0db2-4693-bb09-1ba0e66c3968-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c151ef3-0db2-4693-bb09-1ba0e66c3968" (UID: "4c151ef3-0db2-4693-bb09-1ba0e66c3968"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:50:40.006818 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:40.006790    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"]
Sep 19 11:50:40.008985 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:40.008960    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c151ef3-0db2-4693-bb09-1ba0e66c3968-tls-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:50:40.008985 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:40.008988    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c151ef3-0db2-4693-bb09-1ba0e66c3968-dshm\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:50:40.009154 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:40.009001    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncm6z\" (UniqueName: \"kubernetes.io/projected/4c151ef3-0db2-4693-bb09-1ba0e66c3968-kube-api-access-ncm6z\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:50:40.009154 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:50:40.008965    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38ac71e_49f6_44fb_a3c7_8417fa6c36fe.slice/crio-dc9601c067bc538129b6745f08628906fb7614f37519fe9aea5661a502b912ba WatchSource:0}: Error finding container dc9601c067bc538129b6745f08628906fb7614f37519fe9aea5661a502b912ba: Status 404 returned error can't find the container with id dc9601c067bc538129b6745f08628906fb7614f37519fe9aea5661a502b912ba
Sep 19 11:50:40.411839 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:40.411803    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4" event={"ID":"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe","Type":"ContainerStarted","Data":"dc9601c067bc538129b6745f08628906fb7614f37519fe9aea5661a502b912ba"}
Sep 19 11:50:40.413193 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:40.413164    2584 generic.go:356] "Generic (PLEG): container finished" podID="4c151ef3-0db2-4693-bb09-1ba0e66c3968" containerID="bbe782c7f108ff72f6d7c76035ba49b084bd63b1afd6042b4aa34cfa288bfe17" exitCode=0
Sep 19 11:50:40.413306 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:40.413208    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"
Sep 19 11:50:40.413306 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:40.413233    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht" event={"ID":"4c151ef3-0db2-4693-bb09-1ba0e66c3968","Type":"ContainerDied","Data":"bbe782c7f108ff72f6d7c76035ba49b084bd63b1afd6042b4aa34cfa288bfe17"}
Sep 19 11:50:40.413306 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:40.413273    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht" event={"ID":"4c151ef3-0db2-4693-bb09-1ba0e66c3968","Type":"ContainerDied","Data":"b300bc288e79d8db41f38b96b49643cc65b0e9d646ea9323e83c86921ec47be8"}
Sep 19 11:50:40.413306 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:40.413294    2584 scope.go:117] "RemoveContainer" containerID="bbe782c7f108ff72f6d7c76035ba49b084bd63b1afd6042b4aa34cfa288bfe17"
Sep 19 11:50:40.429875 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:40.429856    2584 scope.go:117] "RemoveContainer" containerID="bbe782c7f108ff72f6d7c76035ba49b084bd63b1afd6042b4aa34cfa288bfe17"
Sep 19 11:50:40.430248 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:50:40.430224    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe782c7f108ff72f6d7c76035ba49b084bd63b1afd6042b4aa34cfa288bfe17\": container with ID starting with bbe782c7f108ff72f6d7c76035ba49b084bd63b1afd6042b4aa34cfa288bfe17 not found: ID does not exist" containerID="bbe782c7f108ff72f6d7c76035ba49b084bd63b1afd6042b4aa34cfa288bfe17"
Sep 19 11:50:40.430310 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:40.430262    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe782c7f108ff72f6d7c76035ba49b084bd63b1afd6042b4aa34cfa288bfe17"} err="failed to get container status \"bbe782c7f108ff72f6d7c76035ba49b084bd63b1afd6042b4aa34cfa288bfe17\": rpc error: code = NotFound desc = could not find container \"bbe782c7f108ff72f6d7c76035ba49b084bd63b1afd6042b4aa34cfa288bfe17\": container with ID starting with bbe782c7f108ff72f6d7c76035ba49b084bd63b1afd6042b4aa34cfa288bfe17 not found: ID does not exist"
Sep 19 11:50:40.455368 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:40.455336    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"]
Sep 19 11:50:40.459393 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:40.459269    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-333f49/test-model-333f49-kserve-5c9647f9c4-6hfht"]
Sep 19 11:50:41.421684 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:41.421643    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4" event={"ID":"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe","Type":"ContainerStarted","Data":"c852ef35efbdf10f2f77cf2329a078dbb258f44a5efd70c366bcea65708d3dfc"}
Sep 19 11:50:41.422143 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:41.421773    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"
Sep 19 11:50:41.439973 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:41.439906    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4" podStartSLOduration=3.047329399 podStartE2EDuration="3.439887811s" totalImagesPullingTime="392.558412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:50:41.437631269 +0000 UTC m=+1025.637661073" watchObservedRunningTime="2026-09-19 11:50:41.439887811 +0000 UTC m=+1025.639917613"
Sep 19 11:50:41.906189 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:41.906153    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c151ef3-0db2-4693-bb09-1ba0e66c3968" path="/var/lib/kubelet/pods/4c151ef3-0db2-4693-bb09-1ba0e66c3968/volumes"
Sep 19 11:50:47.433208 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:47.433174    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"
Sep 19 11:50:54.393113 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:54.393070    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:54.671360 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:54.671275    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:54.772300 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:54.772265    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:55.566664 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:55.566626    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:56.355858 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:56.355820    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:57.142926 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:57.142889    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:57.566746 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:57.566714    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:57.818364 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:57.818269    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:58.661120 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:58.661085    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:59.358340 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:59.358305    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:59.644288 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:59.644196    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:50:59.942959 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:50:59.942842    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:00.209563 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:00.209436    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:00.307968 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:00.307930    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:00.915508 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:00.915475    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:01.748668 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:01.748635    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:02.400767 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:02.400723    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:03.050607 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:03.050570    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:03.961032 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:03.961004    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:04.591127 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:04.591092    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:05.370725 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:05.370684    2584 patch_prober.go:28] interesting pod/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 19 11:51:05.370952 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:05.370748    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b" podUID="138969f5-591e-4ede-8d44-0f568c959000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 19 11:51:05.633712 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:05.633632    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:06.183095 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.183056    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"]
Sep 19 11:51:06.183353 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.183325    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw" podUID="5736df77-9ee0-47f7-b478-00e7a1f0f52d" containerName="maas-api" containerID="cri-o://e187e452e6d5b2dd71d18c44a4b9883402cc790d3dfe1e58c72df55b42b5363d" gracePeriod=30
Sep 19 11:51:06.212445 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.212394    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:06.276164 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.276125    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:06.329883 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.329850    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:06.389302 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.389104    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:06.440163 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.439985    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:06.458374 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.458339    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"
Sep 19 11:51:06.570136 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.570089    2584 generic.go:356] "Generic (PLEG): container finished" podID="5736df77-9ee0-47f7-b478-00e7a1f0f52d" containerID="e187e452e6d5b2dd71d18c44a4b9883402cc790d3dfe1e58c72df55b42b5363d" exitCode=0
Sep 19 11:51:06.570330 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.570153    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"
Sep 19 11:51:06.570330 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.570179    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw" event={"ID":"5736df77-9ee0-47f7-b478-00e7a1f0f52d","Type":"ContainerDied","Data":"e187e452e6d5b2dd71d18c44a4b9883402cc790d3dfe1e58c72df55b42b5363d"}
Sep 19 11:51:06.570330 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.570259    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw" event={"ID":"5736df77-9ee0-47f7-b478-00e7a1f0f52d","Type":"ContainerDied","Data":"aedd62a3e9c5292638d48eb1bfc21b09c088d04c8c88ebfa0b4f1c62266cdf08"}
Sep 19 11:51:06.570330 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.570293    2584 scope.go:117] "RemoveContainer" containerID="e187e452e6d5b2dd71d18c44a4b9883402cc790d3dfe1e58c72df55b42b5363d"
Sep 19 11:51:06.581377 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.581348    2584 scope.go:117] "RemoveContainer" containerID="e187e452e6d5b2dd71d18c44a4b9883402cc790d3dfe1e58c72df55b42b5363d"
Sep 19 11:51:06.581822 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:51:06.581800    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e187e452e6d5b2dd71d18c44a4b9883402cc790d3dfe1e58c72df55b42b5363d\": container with ID starting with e187e452e6d5b2dd71d18c44a4b9883402cc790d3dfe1e58c72df55b42b5363d not found: ID does not exist" containerID="e187e452e6d5b2dd71d18c44a4b9883402cc790d3dfe1e58c72df55b42b5363d"
Sep 19 11:51:06.581962 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.581831    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e187e452e6d5b2dd71d18c44a4b9883402cc790d3dfe1e58c72df55b42b5363d"} err="failed to get container status \"e187e452e6d5b2dd71d18c44a4b9883402cc790d3dfe1e58c72df55b42b5363d\": rpc error: code = NotFound desc = could not find container \"e187e452e6d5b2dd71d18c44a4b9883402cc790d3dfe1e58c72df55b42b5363d\": container with ID starting with e187e452e6d5b2dd71d18c44a4b9883402cc790d3dfe1e58c72df55b42b5363d not found: ID does not exist"
Sep 19 11:51:06.584257 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.584230    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5736df77-9ee0-47f7-b478-00e7a1f0f52d-metrics-certs\") pod \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\" (UID: \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\") "
Sep 19 11:51:06.584384 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.584287    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5736df77-9ee0-47f7-b478-00e7a1f0f52d-openshift-service-ca\") pod \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\" (UID: \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\") "
Sep 19 11:51:06.584456 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.584389    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5736df77-9ee0-47f7-b478-00e7a1f0f52d-maas-api-tls\") pod \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\" (UID: \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\") "
Sep 19 11:51:06.584456 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.584447    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr7sq\" (UniqueName: \"kubernetes.io/projected/5736df77-9ee0-47f7-b478-00e7a1f0f52d-kube-api-access-zr7sq\") pod \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\" (UID: \"5736df77-9ee0-47f7-b478-00e7a1f0f52d\") "
Sep 19 11:51:06.584662 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.584634    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5736df77-9ee0-47f7-b478-00e7a1f0f52d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5736df77-9ee0-47f7-b478-00e7a1f0f52d" (UID: "5736df77-9ee0-47f7-b478-00e7a1f0f52d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:51:06.585179 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.585047    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5736df77-9ee0-47f7-b478-00e7a1f0f52d-openshift-service-ca\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:51:06.587362 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.587313    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5736df77-9ee0-47f7-b478-00e7a1f0f52d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5736df77-9ee0-47f7-b478-00e7a1f0f52d" (UID: "5736df77-9ee0-47f7-b478-00e7a1f0f52d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:51:06.588144 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.588099    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5736df77-9ee0-47f7-b478-00e7a1f0f52d-kube-api-access-zr7sq" (OuterVolumeSpecName: "kube-api-access-zr7sq") pod "5736df77-9ee0-47f7-b478-00e7a1f0f52d" (UID: "5736df77-9ee0-47f7-b478-00e7a1f0f52d"). InnerVolumeSpecName "kube-api-access-zr7sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:51:06.588764 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.588743    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5736df77-9ee0-47f7-b478-00e7a1f0f52d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5736df77-9ee0-47f7-b478-00e7a1f0f52d" (UID: "5736df77-9ee0-47f7-b478-00e7a1f0f52d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:51:06.686197 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.686102    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5736df77-9ee0-47f7-b478-00e7a1f0f52d-metrics-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:51:06.686197 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.686136    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5736df77-9ee0-47f7-b478-00e7a1f0f52d-maas-api-tls\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:51:06.686197 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.686145    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr7sq\" (UniqueName: \"kubernetes.io/projected/5736df77-9ee0-47f7-b478-00e7a1f0f52d-kube-api-access-zr7sq\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:51:06.738817 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.738785    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:06.839876 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.839849    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:06.903864 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.903811    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"]
Sep 19 11:51:06.910589 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:06.910547    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-333f49-86869464f5-5l9kw"]
Sep 19 11:51:07.577489 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:07.577437    2584 generic.go:356] "Generic (PLEG): container finished" podID="138969f5-591e-4ede-8d44-0f568c959000" containerID="744ccd96d9e1198bdb5341c19cc9f6ab05149acf9bf47c125303a76055d4d5a1" exitCode=0
Sep 19 11:51:07.577710 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:07.577519    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b" event={"ID":"138969f5-591e-4ede-8d44-0f568c959000","Type":"ContainerDied","Data":"744ccd96d9e1198bdb5341c19cc9f6ab05149acf9bf47c125303a76055d4d5a1"}
Sep 19 11:51:07.578123 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:07.578095    2584 scope.go:117] "RemoveContainer" containerID="744ccd96d9e1198bdb5341c19cc9f6ab05149acf9bf47c125303a76055d4d5a1"
Sep 19 11:51:07.579360 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:07.579327    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:51:07.579468 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:07.579422    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:51:07.579532 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:07.579470    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:51:07.688632 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:07.688601    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:07.904427 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:07.904340    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5736df77-9ee0-47f7-b478-00e7a1f0f52d" path="/var/lib/kubelet/pods/5736df77-9ee0-47f7-b478-00e7a1f0f52d/volumes"
Sep 19 11:51:08.349402 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:08.349364    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:08.587001 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:08.586959    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b" event={"ID":"138969f5-591e-4ede-8d44-0f568c959000","Type":"ContainerStarted","Data":"a97643bfe6e139d5b53736a8f8d568df7b84013e81d353ef053a735192dc1466"}
Sep 19 11:51:09.085822 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:09.085745    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:51:09.086306 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:09.085989    2584 patch_prober.go:28] interesting pod/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 19 11:51:09.086306 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:09.086042    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b" podUID="138969f5-591e-4ede-8d44-0f568c959000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 19 11:51:09.129277 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:09.129240    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:09.766037 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:09.766006    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:10.086510 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:10.086475    2584 patch_prober.go:28] interesting pod/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 19 11:51:10.086900 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:10.086530    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b" podUID="138969f5-591e-4ede-8d44-0f568c959000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 19 11:51:10.443702 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:10.443620    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:11.085690 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:11.085655    2584 patch_prober.go:28] interesting pod/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 19 11:51:11.085974 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:11.085715    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b" podUID="138969f5-591e-4ede-8d44-0f568c959000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 19 11:51:12.086667 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:12.086624    2584 patch_prober.go:28] interesting pod/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 19 11:51:12.087175 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:12.086685    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b" podUID="138969f5-591e-4ede-8d44-0f568c959000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 19 11:51:12.530035 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:12.529939    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:12.584122 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:12.584093    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:12.778537 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:12.778502    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:13.086601 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:13.086555    2584 patch_prober.go:28] interesting pod/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 19 11:51:13.086800 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:13.086628    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b" podUID="138969f5-591e-4ede-8d44-0f568c959000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 19 11:51:13.384545 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:13.384461    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:14.085623 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:14.085577    2584 patch_prober.go:28] interesting pod/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 19 11:51:14.085810 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:14.085658    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b" podUID="138969f5-591e-4ede-8d44-0f568c959000" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 19 11:51:14.157475 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:14.157433    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:14.788105 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:14.788074    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:15.089739 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:15.089697    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:51:15.090103 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:15.090086    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:51:15.090856 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:15.090832    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:51:15.439877 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:15.439772    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:16.263030 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:16.262999    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:16.893297 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:16.893264    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:17.554468 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:17.554429    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:20.564594 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:20.564562    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:20.619817 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:20.619786    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:23.666098 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:23.666056    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"]
Sep 19 11:51:23.666641 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:23.666368    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx" podUID="be16a33a-bb0c-4330-a9b4-0c2d8c53b04f" containerName="maas-api" containerID="cri-o://403f9aa250e46a81669c6327cacce24c5d43fc8a2ae0806613e2caa30a080fbe" gracePeriod=30
Sep 19 11:51:23.683964 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:23.683714    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:23.749883 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:23.749840    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:23.805087 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:23.805047    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:23.907854 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:23.907822    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:23.930087 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:23.930017    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"
Sep 19 11:51:24.007557 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.007520    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:24.064035 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.064004    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhsnd\" (UniqueName: \"kubernetes.io/projected/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-kube-api-access-hhsnd\") pod \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\" (UID: \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\") "
Sep 19 11:51:24.064221 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.064099    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-openshift-service-ca\") pod \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\" (UID: \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\") "
Sep 19 11:51:24.064221 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.064210    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-maas-api-tls\") pod \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\" (UID: \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\") "
Sep 19 11:51:24.064356 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.064243    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-metrics-certs\") pod \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\" (UID: \"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f\") "
Sep 19 11:51:24.064562 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.064529    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be16a33a-bb0c-4330-a9b4-0c2d8c53b04f" (UID: "be16a33a-bb0c-4330-a9b4-0c2d8c53b04f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:51:24.064658 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.064643    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-openshift-service-ca\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:51:24.067097 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.067069    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "be16a33a-bb0c-4330-a9b4-0c2d8c53b04f" (UID: "be16a33a-bb0c-4330-a9b4-0c2d8c53b04f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:51:24.067097 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.067087    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be16a33a-bb0c-4330-a9b4-0c2d8c53b04f" (UID: "be16a33a-bb0c-4330-a9b4-0c2d8c53b04f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:51:24.067256 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.067182    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-kube-api-access-hhsnd" (OuterVolumeSpecName: "kube-api-access-hhsnd") pod "be16a33a-bb0c-4330-a9b4-0c2d8c53b04f" (UID: "be16a33a-bb0c-4330-a9b4-0c2d8c53b04f"). InnerVolumeSpecName "kube-api-access-hhsnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:51:24.165784 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.165745    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhsnd\" (UniqueName: \"kubernetes.io/projected/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-kube-api-access-hhsnd\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:51:24.165784 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.165781    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-maas-api-tls\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:51:24.165784 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.165792    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f-metrics-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:51:24.303332 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.303242    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:24.366847 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.366815    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:24.696273 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.696237    2584 generic.go:356] "Generic (PLEG): container finished" podID="be16a33a-bb0c-4330-a9b4-0c2d8c53b04f" containerID="403f9aa250e46a81669c6327cacce24c5d43fc8a2ae0806613e2caa30a080fbe" exitCode=0
Sep 19 11:51:24.696758 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.696291    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"
Sep 19 11:51:24.696758 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.696325    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx" event={"ID":"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f","Type":"ContainerDied","Data":"403f9aa250e46a81669c6327cacce24c5d43fc8a2ae0806613e2caa30a080fbe"}
Sep 19 11:51:24.696758 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.696394    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx" event={"ID":"be16a33a-bb0c-4330-a9b4-0c2d8c53b04f","Type":"ContainerDied","Data":"fe98962c496d7765b8df57efef4a52305aeb8d54b05609f36ced791166c09d7d"}
Sep 19 11:51:24.696758 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.696424    2584 scope.go:117] "RemoveContainer" containerID="403f9aa250e46a81669c6327cacce24c5d43fc8a2ae0806613e2caa30a080fbe"
Sep 19 11:51:24.706600 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.706576    2584 scope.go:117] "RemoveContainer" containerID="403f9aa250e46a81669c6327cacce24c5d43fc8a2ae0806613e2caa30a080fbe"
Sep 19 11:51:24.707004 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:51:24.706983    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"403f9aa250e46a81669c6327cacce24c5d43fc8a2ae0806613e2caa30a080fbe\": container with ID starting with 403f9aa250e46a81669c6327cacce24c5d43fc8a2ae0806613e2caa30a080fbe not found: ID does not exist" containerID="403f9aa250e46a81669c6327cacce24c5d43fc8a2ae0806613e2caa30a080fbe"
Sep 19 11:51:24.707074 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.707015    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"403f9aa250e46a81669c6327cacce24c5d43fc8a2ae0806613e2caa30a080fbe"} err="failed to get container status \"403f9aa250e46a81669c6327cacce24c5d43fc8a2ae0806613e2caa30a080fbe\": rpc error: code = NotFound desc = could not find container \"403f9aa250e46a81669c6327cacce24c5d43fc8a2ae0806613e2caa30a080fbe\": container with ID starting with 403f9aa250e46a81669c6327cacce24c5d43fc8a2ae0806613e2caa30a080fbe not found: ID does not exist"
Sep 19 11:51:24.720012 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.719979    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"]
Sep 19 11:51:24.726742 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:24.726703    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d8d8bb-5d4474c6bb-9znkx"]
Sep 19 11:51:25.037657 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:25.037564    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:25.788105 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:25.788061    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:25.904329 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:25.904292    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be16a33a-bb0c-4330-a9b4-0c2d8c53b04f" path="/var/lib/kubelet/pods/be16a33a-bb0c-4330-a9b4-0c2d8c53b04f/volumes"
Sep 19 11:51:26.550677 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:26.550644    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:27.173856 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:27.173825    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:29.929823 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:29.929787    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:29.990685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:29.990648    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:30.181109 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:30.181023    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:30.741374 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:30.741323    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:31.021465 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:31.021382    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:31.221386 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:31.221351    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:33.656441 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:33.656412    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:34.261561 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:34.261531    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:34.369363 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:34.369329    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:34.473102 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:34.473066    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:35.148601 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:35.148566    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:35.976251 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:35.976221    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:36.673821 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:36.673788    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:37.118333 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:37.118300    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:37.338578 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:37.338547    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:37.439019 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:37.438931    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:37.673613 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:37.673572    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:37.798805 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:37.798715    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:37.894234 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:37.894191    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:39.757969 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:39.757906    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:59.189441 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.189397    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:59.244106 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.244056    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"]
Sep 19 11:51:59.244498 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.244484    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c151ef3-0db2-4693-bb09-1ba0e66c3968" containerName="main"
Sep 19 11:51:59.244569 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.244501    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c151ef3-0db2-4693-bb09-1ba0e66c3968" containerName="main"
Sep 19 11:51:59.244569 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.244515    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be16a33a-bb0c-4330-a9b4-0c2d8c53b04f" containerName="maas-api"
Sep 19 11:51:59.244569 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.244521    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="be16a33a-bb0c-4330-a9b4-0c2d8c53b04f" containerName="maas-api"
Sep 19 11:51:59.244569 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.244540    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5736df77-9ee0-47f7-b478-00e7a1f0f52d" containerName="maas-api"
Sep 19 11:51:59.244569 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.244546    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5736df77-9ee0-47f7-b478-00e7a1f0f52d" containerName="maas-api"
Sep 19 11:51:59.244750 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.244607    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5736df77-9ee0-47f7-b478-00e7a1f0f52d" containerName="maas-api"
Sep 19 11:51:59.244750 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.244617    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c151ef3-0db2-4693-bb09-1ba0e66c3968" containerName="main"
Sep 19 11:51:59.244750 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.244624    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="be16a33a-bb0c-4330-a9b4-0c2d8c53b04f" containerName="maas-api"
Sep 19 11:51:59.247614 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.247595    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.250229 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.250199    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7eb15e\"/\"distinct-2-w0-7eb15e-kserve-self-signed-certs\""
Sep 19 11:51:59.255896 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.255869    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"]
Sep 19 11:51:59.279488 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.279450    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:59.334149 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.334116    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:59.422575 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.422542    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-tmp-dir\") pod \"distinct-2-w0-7eb15e-kserve-757594777b-kvv2g\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") " pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.422750 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.422585    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfhgf\" (UniqueName: \"kubernetes.io/projected/7173d3df-0aec-490f-9afa-ed430d9d4a99-kube-api-access-tfhgf\") pod \"distinct-2-w0-7eb15e-kserve-757594777b-kvv2g\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") " pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.422750 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.422612    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7173d3df-0aec-490f-9afa-ed430d9d4a99-tls-certs\") pod \"distinct-2-w0-7eb15e-kserve-757594777b-kvv2g\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") " pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.422750 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.422721    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-model-cache\") pod \"distinct-2-w0-7eb15e-kserve-757594777b-kvv2g\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") " pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.422860 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.422773    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-dshm\") pod \"distinct-2-w0-7eb15e-kserve-757594777b-kvv2g\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") " pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.422860 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.422797    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-home\") pod \"distinct-2-w0-7eb15e-kserve-757594777b-kvv2g\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") " pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.523810 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.523709    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-tmp-dir\") pod \"distinct-2-w0-7eb15e-kserve-757594777b-kvv2g\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") " pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.523810 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.523756    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfhgf\" (UniqueName: \"kubernetes.io/projected/7173d3df-0aec-490f-9afa-ed430d9d4a99-kube-api-access-tfhgf\") pod \"distinct-2-w0-7eb15e-kserve-757594777b-kvv2g\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") " pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.523810 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.523781    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7173d3df-0aec-490f-9afa-ed430d9d4a99-tls-certs\") pod \"distinct-2-w0-7eb15e-kserve-757594777b-kvv2g\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") " pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.524143 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.523822    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-model-cache\") pod \"distinct-2-w0-7eb15e-kserve-757594777b-kvv2g\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") " pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.524143 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.523856    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-dshm\") pod \"distinct-2-w0-7eb15e-kserve-757594777b-kvv2g\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") " pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.524143 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.523873    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-home\") pod \"distinct-2-w0-7eb15e-kserve-757594777b-kvv2g\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") " pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.524303 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.524229    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-tmp-dir\") pod \"distinct-2-w0-7eb15e-kserve-757594777b-kvv2g\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") " pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.524303 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.524251    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-model-cache\") pod \"distinct-2-w0-7eb15e-kserve-757594777b-kvv2g\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") " pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.524397 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.524302    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-home\") pod \"distinct-2-w0-7eb15e-kserve-757594777b-kvv2g\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") " pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.526480 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.526456    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-dshm\") pod \"distinct-2-w0-7eb15e-kserve-757594777b-kvv2g\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") " pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.526769 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.526749    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7173d3df-0aec-490f-9afa-ed430d9d4a99-tls-certs\") pod \"distinct-2-w0-7eb15e-kserve-757594777b-kvv2g\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") " pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.532040 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.532009    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfhgf\" (UniqueName: \"kubernetes.io/projected/7173d3df-0aec-490f-9afa-ed430d9d4a99-kube-api-access-tfhgf\") pod \"distinct-2-w0-7eb15e-kserve-757594777b-kvv2g\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") " pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.559802 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.559762    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:51:59.597887 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.597847    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:51:59.841177 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.841149    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"]
Sep 19 11:51:59.843227 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:51:59.843198    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7173d3df_0aec_490f_9afa_ed430d9d4a99.slice/crio-b24b9bea91bbcc199db0773b3091080721dcdbaef4abe2bbbe30bc6f9e615714 WatchSource:0}: Error finding container b24b9bea91bbcc199db0773b3091080721dcdbaef4abe2bbbe30bc6f9e615714: Status 404 returned error can't find the container with id b24b9bea91bbcc199db0773b3091080721dcdbaef4abe2bbbe30bc6f9e615714
Sep 19 11:51:59.907490 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.907452    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g" event={"ID":"7173d3df-0aec-490f-9afa-ed430d9d4a99","Type":"ContainerStarted","Data":"b24b9bea91bbcc199db0773b3091080721dcdbaef4abe2bbbe30bc6f9e615714"}
Sep 19 11:51:59.908095 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:51:59.908070    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:00.061372 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:00.061335    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:00.183479 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:00.183390    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:00.719076 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:00.719047    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:00.911184 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:00.911147    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g" event={"ID":"7173d3df-0aec-490f-9afa-ed430d9d4a99","Type":"ContainerStarted","Data":"edde79a142c65f6a662bb2e5e402823759f2420b405c68e80044da6381b09b26"}
Sep 19 11:52:00.931768 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:00.931707    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g" podStartSLOduration=1.931688748 podStartE2EDuration="1.931688748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:52:00.930106243 +0000 UTC m=+1105.130136048" watchObservedRunningTime="2026-09-19 11:52:00.931688748 +0000 UTC m=+1105.131718548"
Sep 19 11:52:01.471491 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:01.471452    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:01.560710 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:01.560665    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:52:02.361028 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:02.360995    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:02.654600 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:02.654484    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:02.753382 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:02.753353    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:03.194225 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:03.194184    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:04.027041 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.027008    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"]
Sep 19 11:52:04.030737 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.030712    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"
Sep 19 11:52:04.033746 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.033718    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-4a8a59-serving-cert\""
Sep 19 11:52:04.040687 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.040660    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"]
Sep 19 11:52:04.059223 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.059145    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:04.066731 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.066699    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f967afc0-b956-4227-97fe-a1329c2e544a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q\" (UID: \"f967afc0-b956-4227-97fe-a1329c2e544a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"
Sep 19 11:52:04.066955 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.066760    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f967afc0-b956-4227-97fe-a1329c2e544a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q\" (UID: \"f967afc0-b956-4227-97fe-a1329c2e544a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"
Sep 19 11:52:04.066955 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.066838    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkcdq\" (UniqueName: \"kubernetes.io/projected/f967afc0-b956-4227-97fe-a1329c2e544a-kube-api-access-tkcdq\") pod \"maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q\" (UID: \"f967afc0-b956-4227-97fe-a1329c2e544a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"
Sep 19 11:52:04.066955 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.066878    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f967afc0-b956-4227-97fe-a1329c2e544a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q\" (UID: \"f967afc0-b956-4227-97fe-a1329c2e544a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"
Sep 19 11:52:04.160091 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.160051    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:04.168351 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.168320    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f967afc0-b956-4227-97fe-a1329c2e544a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q\" (UID: \"f967afc0-b956-4227-97fe-a1329c2e544a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"
Sep 19 11:52:04.168483 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.168395    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkcdq\" (UniqueName: \"kubernetes.io/projected/f967afc0-b956-4227-97fe-a1329c2e544a-kube-api-access-tkcdq\") pod \"maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q\" (UID: \"f967afc0-b956-4227-97fe-a1329c2e544a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"
Sep 19 11:52:04.168535 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:52:04.168484    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-serving-cert: secret "maas-api-e2e-shared-b-w3-4a8a59-serving-cert" not found
Sep 19 11:52:04.168575 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.168561    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f967afc0-b956-4227-97fe-a1329c2e544a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q\" (UID: \"f967afc0-b956-4227-97fe-a1329c2e544a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"
Sep 19 11:52:04.168623 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:52:04.168580    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f967afc0-b956-4227-97fe-a1329c2e544a-maas-api-tls podName:f967afc0-b956-4227-97fe-a1329c2e544a nodeName:}" failed. No retries permitted until 2026-09-19 11:52:04.668556995 +0000 UTC m=+1108.868586780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f967afc0-b956-4227-97fe-a1329c2e544a-maas-api-tls") pod "maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q" (UID: "f967afc0-b956-4227-97fe-a1329c2e544a") : secret "maas-api-e2e-shared-b-w3-4a8a59-serving-cert" not found
Sep 19 11:52:04.168686 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.168659    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f967afc0-b956-4227-97fe-a1329c2e544a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q\" (UID: \"f967afc0-b956-4227-97fe-a1329c2e544a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"
Sep 19 11:52:04.169354 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.169325    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f967afc0-b956-4227-97fe-a1329c2e544a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q\" (UID: \"f967afc0-b956-4227-97fe-a1329c2e544a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"
Sep 19 11:52:04.171198 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.171176    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f967afc0-b956-4227-97fe-a1329c2e544a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q\" (UID: \"f967afc0-b956-4227-97fe-a1329c2e544a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"
Sep 19 11:52:04.178118 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.178094    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkcdq\" (UniqueName: \"kubernetes.io/projected/f967afc0-b956-4227-97fe-a1329c2e544a-kube-api-access-tkcdq\") pod \"maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q\" (UID: \"f967afc0-b956-4227-97fe-a1329c2e544a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"
Sep 19 11:52:04.418895 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.418856    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:04.525989 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.525897    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:04.622455 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.622422    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:04.673752 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.673664    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f967afc0-b956-4227-97fe-a1329c2e544a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q\" (UID: \"f967afc0-b956-4227-97fe-a1329c2e544a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"
Sep 19 11:52:04.676495 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.676468    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f967afc0-b956-4227-97fe-a1329c2e544a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q\" (UID: \"f967afc0-b956-4227-97fe-a1329c2e544a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"
Sep 19 11:52:04.945218 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.945116    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"
Sep 19 11:52:04.947106 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:04.947078    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:05.222523 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:05.222495    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"]
Sep 19 11:52:05.225038 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:52:05.225009    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf967afc0_b956_4227_97fe_a1329c2e544a.slice/crio-3f8f0650eb2c86f70182734091ba0b8a28e4b2b05c18f3f918757c0f4e5f3153 WatchSource:0}: Error finding container 3f8f0650eb2c86f70182734091ba0b8a28e4b2b05c18f3f918757c0f4e5f3153: Status 404 returned error can't find the container with id 3f8f0650eb2c86f70182734091ba0b8a28e4b2b05c18f3f918757c0f4e5f3153
Sep 19 11:52:05.766044 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:05.765999    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:05.939187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:05.939084    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q" event={"ID":"f967afc0-b956-4227-97fe-a1329c2e544a","Type":"ContainerStarted","Data":"d5fddbf886e9e2e38c1758028a67401b76858ea3553365c6619a3977ef39f01e"}
Sep 19 11:52:05.939187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:05.939152    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q" event={"ID":"f967afc0-b956-4227-97fe-a1329c2e544a","Type":"ContainerStarted","Data":"3f8f0650eb2c86f70182734091ba0b8a28e4b2b05c18f3f918757c0f4e5f3153"}
Sep 19 11:52:05.939382 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:05.939271    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"
Sep 19 11:52:05.955673 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:05.955617    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q" podStartSLOduration=1.509807 podStartE2EDuration="1.955601921s" totalImagesPullingTime="445.794921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:52:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:52:05.954341752 +0000 UTC m=+1110.154371556" watchObservedRunningTime="2026-09-19 11:52:05.955601921 +0000 UTC m=+1110.155631723"
Sep 19 11:52:06.404228 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:06.404186    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:07.030626 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:07.030587    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:07.776736 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:07.776699    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:09.560701 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:09.560664    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:52:09.573614 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:09.573584    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:52:09.971974 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:09.971863    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:52:11.950268 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:11.950237    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"
Sep 19 11:52:22.417044 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:22.417008    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:25.951435 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:25.951405    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:26.656566 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:26.656517    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:27.282627 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:27.282589    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:27.530358 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:27.530322    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:28.153080 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:28.153043    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:28.795788 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:28.795751    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:29.446078 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:29.446049    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:30.170878 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:30.170839    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:47.044684 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.044623    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"]
Sep 19 11:52:47.049889 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.049863    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.052629 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.052602    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4937ff\"/\"openshift-service-ca.crt\""
Sep 19 11:52:47.053903 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.053883    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4937ff\"/\"kube-root-ca.crt\""
Sep 19 11:52:47.053992 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.053902    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4937ff\"/\"auth-test-model-4937ff-kserve-self-signed-certs\""
Sep 19 11:52:47.053992 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.053946    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4937ff\"/\"default-dockercfg-7njzd\""
Sep 19 11:52:47.058643 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.058612    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"]
Sep 19 11:52:47.106819 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.106784    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-home\") pod \"auth-test-model-4937ff-kserve-59889cbcf6-lvfjn\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.107037 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.106856    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-tls-certs\") pod \"auth-test-model-4937ff-kserve-59889cbcf6-lvfjn\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.107037 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.106887    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-model-cache\") pod \"auth-test-model-4937ff-kserve-59889cbcf6-lvfjn\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.107037 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.106964    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-tmp-dir\") pod \"auth-test-model-4937ff-kserve-59889cbcf6-lvfjn\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.107037 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.107006    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-dshm\") pod \"auth-test-model-4937ff-kserve-59889cbcf6-lvfjn\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.107203 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.107046    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmfg9\" (UniqueName: \"kubernetes.io/projected/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-kube-api-access-zmfg9\") pod \"auth-test-model-4937ff-kserve-59889cbcf6-lvfjn\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.207939 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.207881    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-tls-certs\") pod \"auth-test-model-4937ff-kserve-59889cbcf6-lvfjn\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.208149 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.207958    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-model-cache\") pod \"auth-test-model-4937ff-kserve-59889cbcf6-lvfjn\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.208149 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.208029    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-tmp-dir\") pod \"auth-test-model-4937ff-kserve-59889cbcf6-lvfjn\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.208149 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.208083    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-dshm\") pod \"auth-test-model-4937ff-kserve-59889cbcf6-lvfjn\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.208149 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.208138    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmfg9\" (UniqueName: \"kubernetes.io/projected/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-kube-api-access-zmfg9\") pod \"auth-test-model-4937ff-kserve-59889cbcf6-lvfjn\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.208366 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.208203    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-home\") pod \"auth-test-model-4937ff-kserve-59889cbcf6-lvfjn\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.208498 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.208466    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-model-cache\") pod \"auth-test-model-4937ff-kserve-59889cbcf6-lvfjn\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.208642 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.208501    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-tmp-dir\") pod \"auth-test-model-4937ff-kserve-59889cbcf6-lvfjn\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.209246 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.209219    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-home\") pod \"auth-test-model-4937ff-kserve-59889cbcf6-lvfjn\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.210829 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.210806    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-dshm\") pod \"auth-test-model-4937ff-kserve-59889cbcf6-lvfjn\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.211144 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.211121    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-tls-certs\") pod \"auth-test-model-4937ff-kserve-59889cbcf6-lvfjn\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.219597 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.219571    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmfg9\" (UniqueName: \"kubernetes.io/projected/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-kube-api-access-zmfg9\") pod \"auth-test-model-4937ff-kserve-59889cbcf6-lvfjn\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.238602 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.238560    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:47.361131 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.361091    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:47.532416 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.532387    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:47.637075 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:47.637047    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"]
Sep 19 11:52:47.638907 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:52:47.638878    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8bf92f3_1293_4d69_b8f9_4aa49aa924be.slice/crio-5d788928a897956c532320cd167b0ab2d7abf7539577a37aefff64e983185f9c WatchSource:0}: Error finding container 5d788928a897956c532320cd167b0ab2d7abf7539577a37aefff64e983185f9c: Status 404 returned error can't find the container with id 5d788928a897956c532320cd167b0ab2d7abf7539577a37aefff64e983185f9c
Sep 19 11:52:48.189995 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:48.189945    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn" event={"ID":"f8bf92f3-1293-4d69-b8f9-4aa49aa924be","Type":"ContainerStarted","Data":"b33d3cb8f352b155017541160f2d5f734c0d1ea9fd2df0b0b68c0f9362611181"}
Sep 19 11:52:48.190418 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:48.190007    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn" event={"ID":"f8bf92f3-1293-4d69-b8f9-4aa49aa924be","Type":"ContainerStarted","Data":"5d788928a897956c532320cd167b0ab2d7abf7539577a37aefff64e983185f9c"}
Sep 19 11:52:48.209334 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:48.209252    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn" podStartSLOduration=1.20922716 podStartE2EDuration="1.20922716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:52:48.206154753 +0000 UTC m=+1152.406184570" watchObservedRunningTime="2026-09-19 11:52:48.20922716 +0000 UTC m=+1152.409256962"
Sep 19 11:52:48.362245 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:48.362206    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:48.441279 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:48.441159    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:49.174315 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:49.174282    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:49.854802 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:49.854768    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:50.570877 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:50.570848    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:51.315755 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:51.315719    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:51.989127 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:51.989096    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:52.738962 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:52.738931    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:52:57.361401 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:57.361365    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:57.374299 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:57.374271    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:52:58.265268 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:52:58.265237    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:53:02.844989 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:02.844937    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"]
Sep 19 11:53:02.846115 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:02.846080    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4" podUID="f38ac71e-49f6-44fb-a3c7-8417fa6c36fe" containerName="maas-api" containerID="cri-o://c852ef35efbdf10f2f77cf2329a078dbb258f44a5efd70c366bcea65708d3dfc" gracePeriod=30
Sep 19 11:53:02.879046 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:02.879005    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:02.949534 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:02.949504    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:03.051769 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.051726    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:03.108338 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.108308    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:03.113893 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.113867    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"
Sep 19 11:53:03.167490 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.167456    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-openshift-service-ca\") pod \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\" (UID: \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\") "
Sep 19 11:53:03.167690 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.167498    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-maas-api-tls\") pod \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\" (UID: \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\") "
Sep 19 11:53:03.167690 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.167521    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5k9d\" (UniqueName: \"kubernetes.io/projected/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-kube-api-access-m5k9d\") pod \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\" (UID: \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\") "
Sep 19 11:53:03.167690 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.167592    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-metrics-certs\") pod \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\" (UID: \"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe\") "
Sep 19 11:53:03.167905 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.167858    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f38ac71e-49f6-44fb-a3c7-8417fa6c36fe" (UID: "f38ac71e-49f6-44fb-a3c7-8417fa6c36fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:53:03.169902 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.169872    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f38ac71e-49f6-44fb-a3c7-8417fa6c36fe" (UID: "f38ac71e-49f6-44fb-a3c7-8417fa6c36fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:53:03.170048 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.169892    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f38ac71e-49f6-44fb-a3c7-8417fa6c36fe" (UID: "f38ac71e-49f6-44fb-a3c7-8417fa6c36fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:53:03.170094 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.170053    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-kube-api-access-m5k9d" (OuterVolumeSpecName: "kube-api-access-m5k9d") pod "f38ac71e-49f6-44fb-a3c7-8417fa6c36fe" (UID: "f38ac71e-49f6-44fb-a3c7-8417fa6c36fe"). InnerVolumeSpecName "kube-api-access-m5k9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:53:03.268650 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.268612    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-openshift-service-ca\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:53:03.268650 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.268646    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-maas-api-tls\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:53:03.268650 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.268655    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5k9d\" (UniqueName: \"kubernetes.io/projected/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-kube-api-access-m5k9d\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:53:03.268903 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.268665    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe-metrics-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:53:03.281724 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.281690    2584 generic.go:356] "Generic (PLEG): container finished" podID="f38ac71e-49f6-44fb-a3c7-8417fa6c36fe" containerID="c852ef35efbdf10f2f77cf2329a078dbb258f44a5efd70c366bcea65708d3dfc" exitCode=0
Sep 19 11:53:03.281903 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.281745    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"
Sep 19 11:53:03.281903 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.281780    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4" event={"ID":"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe","Type":"ContainerDied","Data":"c852ef35efbdf10f2f77cf2329a078dbb258f44a5efd70c366bcea65708d3dfc"}
Sep 19 11:53:03.281903 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.281840    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4" event={"ID":"f38ac71e-49f6-44fb-a3c7-8417fa6c36fe","Type":"ContainerDied","Data":"dc9601c067bc538129b6745f08628906fb7614f37519fe9aea5661a502b912ba"}
Sep 19 11:53:03.281903 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.281854    2584 scope.go:117] "RemoveContainer" containerID="c852ef35efbdf10f2f77cf2329a078dbb258f44a5efd70c366bcea65708d3dfc"
Sep 19 11:53:03.295326 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.295297    2584 scope.go:117] "RemoveContainer" containerID="c852ef35efbdf10f2f77cf2329a078dbb258f44a5efd70c366bcea65708d3dfc"
Sep 19 11:53:03.295743 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:53:03.295704    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c852ef35efbdf10f2f77cf2329a078dbb258f44a5efd70c366bcea65708d3dfc\": container with ID starting with c852ef35efbdf10f2f77cf2329a078dbb258f44a5efd70c366bcea65708d3dfc not found: ID does not exist" containerID="c852ef35efbdf10f2f77cf2329a078dbb258f44a5efd70c366bcea65708d3dfc"
Sep 19 11:53:03.295827 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.295756    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c852ef35efbdf10f2f77cf2329a078dbb258f44a5efd70c366bcea65708d3dfc"} err="failed to get container status \"c852ef35efbdf10f2f77cf2329a078dbb258f44a5efd70c366bcea65708d3dfc\": rpc error: code = NotFound desc = could not find container \"c852ef35efbdf10f2f77cf2329a078dbb258f44a5efd70c366bcea65708d3dfc\": container with ID starting with c852ef35efbdf10f2f77cf2329a078dbb258f44a5efd70c366bcea65708d3dfc not found: ID does not exist"
Sep 19 11:53:03.304947 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.304900    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"]
Sep 19 11:53:03.308488 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.308460    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fcfc8fcc-9c8ddf4d9-4mxh4"]
Sep 19 11:53:03.543520 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.543437    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:03.904155 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:03.904120    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f38ac71e-49f6-44fb-a3c7-8417fa6c36fe" path="/var/lib/kubelet/pods/f38ac71e-49f6-44fb-a3c7-8417fa6c36fe/volumes"
Sep 19 11:53:04.254425 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:04.254310    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:05.050833 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:05.050778    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:05.675339 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:05.675294    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:06.147680 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:06.147644    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:06.291294 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:06.291241    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:07.072803 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:07.072764    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:09.629957 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.629890    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"]
Sep 19 11:53:09.630442 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.630342    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f38ac71e-49f6-44fb-a3c7-8417fa6c36fe" containerName="maas-api"
Sep 19 11:53:09.630442 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.630355    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f38ac71e-49f6-44fb-a3c7-8417fa6c36fe" containerName="maas-api"
Sep 19 11:53:09.630442 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.630430    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f38ac71e-49f6-44fb-a3c7-8417fa6c36fe" containerName="maas-api"
Sep 19 11:53:09.634729 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.634705    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:09.637591 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.637569    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4a8a59\"/\"kube-root-ca.crt\""
Sep 19 11:53:09.638961 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.638941    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4a8a59\"/\"default-dockercfg-7wptg\""
Sep 19 11:53:09.639088 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.638967    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4a8a59\"/\"openshift-service-ca.crt\""
Sep 19 11:53:09.639088 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.639026    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4a8a59\"/\"auth-test-model-4a8a59-kserve-self-signed-certs\""
Sep 19 11:53:09.643623 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.643594    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"]
Sep 19 11:53:09.646992 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.646964    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:09.733600 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.733543    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-model-cache\") pod \"auth-test-model-4a8a59-kserve-74dffd8d8-qlql4\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:09.733600 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.733604    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-home\") pod \"auth-test-model-4a8a59-kserve-74dffd8d8-qlql4\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:09.733839 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.733715    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e11958f3-990d-45af-9a32-ae08291ec3e2-tls-certs\") pod \"auth-test-model-4a8a59-kserve-74dffd8d8-qlql4\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:09.733839 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.733776    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4q8\" (UniqueName: \"kubernetes.io/projected/e11958f3-990d-45af-9a32-ae08291ec3e2-kube-api-access-tn4q8\") pod \"auth-test-model-4a8a59-kserve-74dffd8d8-qlql4\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:09.733839 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.733797    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-dshm\") pod \"auth-test-model-4a8a59-kserve-74dffd8d8-qlql4\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:09.733839 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.733821    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-tmp-dir\") pod \"auth-test-model-4a8a59-kserve-74dffd8d8-qlql4\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:09.834420 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.834377    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-model-cache\") pod \"auth-test-model-4a8a59-kserve-74dffd8d8-qlql4\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:09.834420 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.834425    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-home\") pod \"auth-test-model-4a8a59-kserve-74dffd8d8-qlql4\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:09.834653 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.834494    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e11958f3-990d-45af-9a32-ae08291ec3e2-tls-certs\") pod \"auth-test-model-4a8a59-kserve-74dffd8d8-qlql4\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:09.834709 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.834671    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4q8\" (UniqueName: \"kubernetes.io/projected/e11958f3-990d-45af-9a32-ae08291ec3e2-kube-api-access-tn4q8\") pod \"auth-test-model-4a8a59-kserve-74dffd8d8-qlql4\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:09.834752 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.834731    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-dshm\") pod \"auth-test-model-4a8a59-kserve-74dffd8d8-qlql4\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:09.834869 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.834769    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-tmp-dir\") pod \"auth-test-model-4a8a59-kserve-74dffd8d8-qlql4\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:09.834957 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.834899    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-model-cache\") pod \"auth-test-model-4a8a59-kserve-74dffd8d8-qlql4\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:09.835016 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.834959    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-home\") pod \"auth-test-model-4a8a59-kserve-74dffd8d8-qlql4\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:09.835307 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.835271    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-tmp-dir\") pod \"auth-test-model-4a8a59-kserve-74dffd8d8-qlql4\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:09.838031 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.837998    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e11958f3-990d-45af-9a32-ae08291ec3e2-tls-certs\") pod \"auth-test-model-4a8a59-kserve-74dffd8d8-qlql4\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:09.838201 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.838177    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-dshm\") pod \"auth-test-model-4a8a59-kserve-74dffd8d8-qlql4\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:09.844537 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.844174    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4q8\" (UniqueName: \"kubernetes.io/projected/e11958f3-990d-45af-9a32-ae08291ec3e2-kube-api-access-tn4q8\") pod \"auth-test-model-4a8a59-kserve-74dffd8d8-qlql4\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:09.944809 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:09.944706    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:10.219404 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:10.219322    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"]
Sep 19 11:53:10.222942 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:53:10.222882    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode11958f3_990d_45af_9a32_ae08291ec3e2.slice/crio-dbb763a024ff1ed6b8348e6e66c60b2a9002ecc8d00f9dee6009af548526b4d4 WatchSource:0}: Error finding container dbb763a024ff1ed6b8348e6e66c60b2a9002ecc8d00f9dee6009af548526b4d4: Status 404 returned error can't find the container with id dbb763a024ff1ed6b8348e6e66c60b2a9002ecc8d00f9dee6009af548526b4d4
Sep 19 11:53:10.324602 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:10.324565    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4" event={"ID":"e11958f3-990d-45af-9a32-ae08291ec3e2","Type":"ContainerStarted","Data":"9b97c1a215cd6e56dff44b52ed88c336ee679bd891245f4025f7e3469e95583e"}
Sep 19 11:53:10.324785 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:10.324614    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4" event={"ID":"e11958f3-990d-45af-9a32-ae08291ec3e2","Type":"ContainerStarted","Data":"dbb763a024ff1ed6b8348e6e66c60b2a9002ecc8d00f9dee6009af548526b4d4"}
Sep 19 11:53:10.341420 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:10.341355    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4" podStartSLOduration=1.341337458 podStartE2EDuration="1.341337458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:53:10.339475471 +0000 UTC m=+1174.539505288" watchObservedRunningTime="2026-09-19 11:53:10.341337458 +0000 UTC m=+1174.541367260"
Sep 19 11:53:10.945528 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:10.945482    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:11.171187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:11.171149    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:19.945451 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:19.945407    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:19.960159 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:19.960126    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:20.391065 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:20.391031    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:53:21.337809 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:53:21.337763    2584 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b233060_ef58_4c7c_a92c_a39ffd617014.slice/crio-a5938c0b26a0a00808b5f4cab92caa38017a11e3c886cba25311e42acfe5d592.scope\": RecentStats: unable to find data in memory cache]"
Sep 19 11:53:22.393478 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:22.393442    2584 generic.go:356] "Generic (PLEG): container finished" podID="9b233060-ef58-4c7c-a92c-a39ffd617014" containerID="a5938c0b26a0a00808b5f4cab92caa38017a11e3c886cba25311e42acfe5d592" exitCode=137
Sep 19 11:53:22.393885 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:22.393512    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg" event={"ID":"9b233060-ef58-4c7c-a92c-a39ffd617014","Type":"ContainerDied","Data":"a5938c0b26a0a00808b5f4cab92caa38017a11e3c886cba25311e42acfe5d592"}
Sep 19 11:53:22.393959 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:22.393950    2584 scope.go:117] "RemoveContainer" containerID="a5938c0b26a0a00808b5f4cab92caa38017a11e3c886cba25311e42acfe5d592"
Sep 19 11:53:23.401834 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:23.401798    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg" event={"ID":"9b233060-ef58-4c7c-a92c-a39ffd617014","Type":"ContainerStarted","Data":"309bc0daeb4a81484072fd606af98bf4e77ce7fd677fcb1ae5d63e8e2ac58021"}
Sep 19 11:53:23.402333 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:23.402040    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"
Sep 19 11:53:28.194768 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:28.194728    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:28.248613 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:28.248578    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:34.410650 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:34.410615    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"
Sep 19 11:53:37.937082 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:37.937049    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/1.log"
Sep 19 11:53:37.937539 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:37.937366    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/1.log"
Sep 19 11:53:37.943019 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:37.942997    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6th98_2fed27e0-833f-49a8-b11e-8f61df6e6bd5/ovn-acl-logging/0.log"
Sep 19 11:53:37.943412 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:37.943389    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6th98_2fed27e0-833f-49a8-b11e-8f61df6e6bd5/ovn-acl-logging/0.log"
Sep 19 11:53:47.770095 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:47.770054    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:49.113337 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:49.113306    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:53.476759 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:53.476726    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:53.577700 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:53.577665    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:53.717214 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:53.717180    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"]
Sep 19 11:53:53.717423 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:53.717400    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b" podUID="138969f5-591e-4ede-8d44-0f568c959000" containerName="istio-proxy" containerID="cri-o://a97643bfe6e139d5b53736a8f8d568df7b84013e81d353ef053a735192dc1466" gracePeriod=30
Sep 19 11:53:53.735486 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:53.735409    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:54.977242 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:54.977210    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:55.077177 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:55.077138    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:55.177300 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:55.177264    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:55.760936 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:55.760882    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:55.868991 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:55.868950    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:55.963954 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:55.963896    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:57.209260 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:57.209227    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:53:58.984015 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:58.983951    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:53:59.064008 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.063976    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-credential-socket\") pod \"138969f5-591e-4ede-8d44-0f568c959000\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") "
Sep 19 11:53:59.064209 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.064020    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138969f5-591e-4ede-8d44-0f568c959000-istiod-ca-cert\") pod \"138969f5-591e-4ede-8d44-0f568c959000\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") "
Sep 19 11:53:59.064209 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.064046    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-workload-certs\") pod \"138969f5-591e-4ede-8d44-0f568c959000\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") "
Sep 19 11:53:59.064209 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.064088    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-workload-socket\") pod \"138969f5-591e-4ede-8d44-0f568c959000\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") "
Sep 19 11:53:59.064209 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.064125    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138969f5-591e-4ede-8d44-0f568c959000-istio-podinfo\") pod \"138969f5-591e-4ede-8d44-0f568c959000\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") "
Sep 19 11:53:59.064439 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.064230    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138969f5-591e-4ede-8d44-0f568c959000-istio-token\") pod \"138969f5-591e-4ede-8d44-0f568c959000\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") "
Sep 19 11:53:59.064439 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.064284    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-istio-data\") pod \"138969f5-591e-4ede-8d44-0f568c959000\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") "
Sep 19 11:53:59.064439 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.064327    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-istio-envoy\") pod \"138969f5-591e-4ede-8d44-0f568c959000\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") "
Sep 19 11:53:59.064439 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.064369    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lv6t\" (UniqueName: \"kubernetes.io/projected/138969f5-591e-4ede-8d44-0f568c959000-kube-api-access-7lv6t\") pod \"138969f5-591e-4ede-8d44-0f568c959000\" (UID: \"138969f5-591e-4ede-8d44-0f568c959000\") "
Sep 19 11:53:59.064439 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.064348    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "138969f5-591e-4ede-8d44-0f568c959000" (UID: "138969f5-591e-4ede-8d44-0f568c959000"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:53:59.064439 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.064410    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "138969f5-591e-4ede-8d44-0f568c959000" (UID: "138969f5-591e-4ede-8d44-0f568c959000"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:53:59.064760 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.064481    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/138969f5-591e-4ede-8d44-0f568c959000-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "138969f5-591e-4ede-8d44-0f568c959000" (UID: "138969f5-591e-4ede-8d44-0f568c959000"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:53:59.064760 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.064576    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-istio-data" (OuterVolumeSpecName: "istio-data") pod "138969f5-591e-4ede-8d44-0f568c959000" (UID: "138969f5-591e-4ede-8d44-0f568c959000"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:53:59.064760 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.064731    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "138969f5-591e-4ede-8d44-0f568c959000" (UID: "138969f5-591e-4ede-8d44-0f568c959000"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:53:59.064989 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.064813    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-credential-socket\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:53:59.064989 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.064828    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138969f5-591e-4ede-8d44-0f568c959000-istiod-ca-cert\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:53:59.064989 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.064837    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-workload-socket\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:53:59.064989 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.064845    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-istio-data\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:53:59.066685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.066652    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138969f5-591e-4ede-8d44-0f568c959000-kube-api-access-7lv6t" (OuterVolumeSpecName: "kube-api-access-7lv6t") pod "138969f5-591e-4ede-8d44-0f568c959000" (UID: "138969f5-591e-4ede-8d44-0f568c959000"). InnerVolumeSpecName "kube-api-access-7lv6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:53:59.066806 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.066712    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138969f5-591e-4ede-8d44-0f568c959000-istio-token" (OuterVolumeSpecName: "istio-token") pod "138969f5-591e-4ede-8d44-0f568c959000" (UID: "138969f5-591e-4ede-8d44-0f568c959000"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:53:59.066806 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.066713    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/138969f5-591e-4ede-8d44-0f568c959000-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "138969f5-591e-4ede-8d44-0f568c959000" (UID: "138969f5-591e-4ede-8d44-0f568c959000"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 11:53:59.066806 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.066726    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "138969f5-591e-4ede-8d44-0f568c959000" (UID: "138969f5-591e-4ede-8d44-0f568c959000"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:53:59.166208 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.166170    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-istio-envoy\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:53:59.166208 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.166206    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lv6t\" (UniqueName: \"kubernetes.io/projected/138969f5-591e-4ede-8d44-0f568c959000-kube-api-access-7lv6t\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:53:59.166208 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.166217    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138969f5-591e-4ede-8d44-0f568c959000-workload-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:53:59.166446 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.166225    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138969f5-591e-4ede-8d44-0f568c959000-istio-podinfo\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:53:59.166446 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.166237    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138969f5-591e-4ede-8d44-0f568c959000-istio-token\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:53:59.609787 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.609756    2584 generic.go:356] "Generic (PLEG): container finished" podID="138969f5-591e-4ede-8d44-0f568c959000" containerID="a97643bfe6e139d5b53736a8f8d568df7b84013e81d353ef053a735192dc1466" exitCode=0
Sep 19 11:53:59.610000 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.609803    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"
Sep 19 11:53:59.610000 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.609843    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b" event={"ID":"138969f5-591e-4ede-8d44-0f568c959000","Type":"ContainerDied","Data":"a97643bfe6e139d5b53736a8f8d568df7b84013e81d353ef053a735192dc1466"}
Sep 19 11:53:59.610000 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.609897    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b" event={"ID":"138969f5-591e-4ede-8d44-0f568c959000","Type":"ContainerDied","Data":"b1b96b900c3d88cec2693d79454db76ce3465f0d8dcaef5a9743663131b20ab3"}
Sep 19 11:53:59.610000 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.609944    2584 scope.go:117] "RemoveContainer" containerID="a97643bfe6e139d5b53736a8f8d568df7b84013e81d353ef053a735192dc1466"
Sep 19 11:53:59.620407 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.620388    2584 scope.go:117] "RemoveContainer" containerID="744ccd96d9e1198bdb5341c19cc9f6ab05149acf9bf47c125303a76055d4d5a1"
Sep 19 11:53:59.629757 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.629742    2584 scope.go:117] "RemoveContainer" containerID="a97643bfe6e139d5b53736a8f8d568df7b84013e81d353ef053a735192dc1466"
Sep 19 11:53:59.630120 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:53:59.630097    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a97643bfe6e139d5b53736a8f8d568df7b84013e81d353ef053a735192dc1466\": container with ID starting with a97643bfe6e139d5b53736a8f8d568df7b84013e81d353ef053a735192dc1466 not found: ID does not exist" containerID="a97643bfe6e139d5b53736a8f8d568df7b84013e81d353ef053a735192dc1466"
Sep 19 11:53:59.630204 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.630129    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a97643bfe6e139d5b53736a8f8d568df7b84013e81d353ef053a735192dc1466"} err="failed to get container status \"a97643bfe6e139d5b53736a8f8d568df7b84013e81d353ef053a735192dc1466\": rpc error: code = NotFound desc = could not find container \"a97643bfe6e139d5b53736a8f8d568df7b84013e81d353ef053a735192dc1466\": container with ID starting with a97643bfe6e139d5b53736a8f8d568df7b84013e81d353ef053a735192dc1466 not found: ID does not exist"
Sep 19 11:53:59.630204 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.630148    2584 scope.go:117] "RemoveContainer" containerID="744ccd96d9e1198bdb5341c19cc9f6ab05149acf9bf47c125303a76055d4d5a1"
Sep 19 11:53:59.630400 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:53:59.630382    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744ccd96d9e1198bdb5341c19cc9f6ab05149acf9bf47c125303a76055d4d5a1\": container with ID starting with 744ccd96d9e1198bdb5341c19cc9f6ab05149acf9bf47c125303a76055d4d5a1 not found: ID does not exist" containerID="744ccd96d9e1198bdb5341c19cc9f6ab05149acf9bf47c125303a76055d4d5a1"
Sep 19 11:53:59.630463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.630406    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744ccd96d9e1198bdb5341c19cc9f6ab05149acf9bf47c125303a76055d4d5a1"} err="failed to get container status \"744ccd96d9e1198bdb5341c19cc9f6ab05149acf9bf47c125303a76055d4d5a1\": rpc error: code = NotFound desc = could not find container \"744ccd96d9e1198bdb5341c19cc9f6ab05149acf9bf47c125303a76055d4d5a1\": container with ID starting with 744ccd96d9e1198bdb5341c19cc9f6ab05149acf9bf47c125303a76055d4d5a1 not found: ID does not exist"
Sep 19 11:53:59.631183 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.631161    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"]
Sep 19 11:53:59.634419 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.634396    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fcfc8fcc-openshift-default-594cf9cd6d-vnn6b"]
Sep 19 11:53:59.904393 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:53:59.904302    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="138969f5-591e-4ede-8d44-0f568c959000" path="/var/lib/kubelet/pods/138969f5-591e-4ede-8d44-0f568c959000/volumes"
Sep 19 11:54:03.005792 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:03.005760    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:03.060719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:03.060687    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:03.666425 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:03.666397    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:09.397759 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:09.397714    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:09.462182 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:09.462142    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:09.536423 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:09.536394    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:09.636928 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:09.636889    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:10.344494 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:10.344465    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:20.080240 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:20.080199    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:20.301712 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:20.301676    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:41.204540 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:41.204503    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:43.258110 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:43.258072    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:47.025022 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.024974    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:47.091111 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.091076    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:47.375808 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.375771    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:47.633308 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.633206    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"]
Sep 19 11:54:47.633732 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.633710    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="138969f5-591e-4ede-8d44-0f568c959000" containerName="istio-proxy"
Sep 19 11:54:47.633732 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.633730    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="138969f5-591e-4ede-8d44-0f568c959000" containerName="istio-proxy"
Sep 19 11:54:47.633884 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.633827    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="138969f5-591e-4ede-8d44-0f568c959000" containerName="istio-proxy"
Sep 19 11:54:47.637099 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.637079    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.639801 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.639777    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3c21ca-openshift-default-dockercfg-tw2qh\""
Sep 19 11:54:47.648053 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.648020    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"]
Sep 19 11:54:47.658302 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.658265    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:47.753018 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.752981    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839984cb-3e33-4182-b323-751344047d86-istio-podinfo\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.753215 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.753035    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-workload-socket\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.753215 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.753069    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wghl6\" (UniqueName: \"kubernetes.io/projected/839984cb-3e33-4182-b323-751344047d86-kube-api-access-wghl6\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.753215 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.753108    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-istio-envoy\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.753215 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.753127    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/839984cb-3e33-4182-b323-751344047d86-istio-token\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.753374 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.753212    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-workload-certs\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.753374 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.753254    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-istio-data\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.753374 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.753274    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-credential-socket\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.753374 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.753303    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839984cb-3e33-4182-b323-751344047d86-istiod-ca-cert\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.854855 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.854808    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-istio-envoy\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.854855 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.854856    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/839984cb-3e33-4182-b323-751344047d86-istio-token\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.855121 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.854904    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-workload-certs\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.855121 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.855052    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-istio-data\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.855121 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.855083    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-credential-socket\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.855242 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.855123    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839984cb-3e33-4182-b323-751344047d86-istiod-ca-cert\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.855242 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.855176    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839984cb-3e33-4182-b323-751344047d86-istio-podinfo\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.855242 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.855226    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-workload-socket\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.855416 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.855358    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wghl6\" (UniqueName: \"kubernetes.io/projected/839984cb-3e33-4182-b323-751344047d86-kube-api-access-wghl6\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.855416 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.855398    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-workload-certs\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.855514 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.855439    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-istio-data\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.855514 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.855477    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-credential-socket\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.855758 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.855679    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-workload-socket\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.856025 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.856004    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839984cb-3e33-4182-b323-751344047d86-istiod-ca-cert\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.857461 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.857437    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-istio-envoy\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.857889 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.857870    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839984cb-3e33-4182-b323-751344047d86-istio-podinfo\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.862997 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.862964    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/839984cb-3e33-4182-b323-751344047d86-istio-token\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.863116 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.863004    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wghl6\" (UniqueName: \"kubernetes.io/projected/839984cb-3e33-4182-b323-751344047d86-kube-api-access-wghl6\") pod \"e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") " pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:47.945960 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.945847    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:47.948831 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:47.948813    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:48.229054 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:48.229017    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"]
Sep 19 11:54:48.233425 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:54:48.233384    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839984cb_3e33_4182_b323_751344047d86.slice/crio-e33495c78093749228efd42243e803f501051c3b5fa355d954ebfc5102723de0 WatchSource:0}: Error finding container e33495c78093749228efd42243e803f501051c3b5fa355d954ebfc5102723de0: Status 404 returned error can't find the container with id e33495c78093749228efd42243e803f501051c3b5fa355d954ebfc5102723de0
Sep 19 11:54:48.236229 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:48.236193    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:54:48.236322 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:48.236268    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:54:48.236322 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:48.236298    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:54:48.895396 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:48.895360    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx" event={"ID":"839984cb-3e33-4182-b323-751344047d86","Type":"ContainerStarted","Data":"cb0a35960a13be190c1b439931a9de2b249c98d6bf8eebcd4dda8c008b881cb6"}
Sep 19 11:54:48.895626 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:48.895423    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx" event={"ID":"839984cb-3e33-4182-b323-751344047d86","Type":"ContainerStarted","Data":"e33495c78093749228efd42243e803f501051c3b5fa355d954ebfc5102723de0"}
Sep 19 11:54:48.918965 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:48.918885    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx" podStartSLOduration=1.918863567 podStartE2EDuration="1.918863567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:54:48.914642724 +0000 UTC m=+1273.114672539" watchObservedRunningTime="2026-09-19 11:54:48.918863567 +0000 UTC m=+1273.118893386"
Sep 19 11:54:48.949221 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:48.949162    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:49.955957 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:49.955904    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:50.232517 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:50.232425    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:50.895993 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:50.895960    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:50.906608 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:50.906583    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:50.907659 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:50.907639    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:54:50.995828 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:50.995793    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:51.683459 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:51.683424    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:52.350221 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:52.350184    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:53.052417 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:53.052388    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:53.679281 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:53.679247    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:54.564071 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:54.564039    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:54.665675 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:54.665642    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:54.988305 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:54.988202    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:55.081488 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:55.081449    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:54:55.146876 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:54:55.146848    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:17.253125 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:17.253089    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:17.882347 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:17.882292    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:18.066659 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:18.066619    2584 generic.go:356] "Generic (PLEG): container finished" podID="839984cb-3e33-4182-b323-751344047d86" containerID="cb0a35960a13be190c1b439931a9de2b249c98d6bf8eebcd4dda8c008b881cb6" exitCode=0
Sep 19 11:55:18.066870 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:18.066694    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx" event={"ID":"839984cb-3e33-4182-b323-751344047d86","Type":"ContainerDied","Data":"cb0a35960a13be190c1b439931a9de2b249c98d6bf8eebcd4dda8c008b881cb6"}
Sep 19 11:55:18.067241 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:18.067222    2584 scope.go:117] "RemoveContainer" containerID="cb0a35960a13be190c1b439931a9de2b249c98d6bf8eebcd4dda8c008b881cb6"
Sep 19 11:55:18.068454 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:18.068424    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:55:18.068552 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:18.068532    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:55:18.068617 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:18.068590    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:55:18.496468 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:18.496362    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:19.074883 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:19.074845    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx" event={"ID":"839984cb-3e33-4182-b323-751344047d86","Type":"ContainerStarted","Data":"85b24fce01ee96b7bbe22547f563c6102fe1a737c6032ac4b0819af713fe97e7"}
Sep 19 11:55:19.260460 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:19.260428    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:19.893232 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:19.893200    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:19.949377 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:19.949346    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:55:19.950781 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:19.950754    2584 patch_prober.go:28] interesting pod/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 19 11:55:19.950932 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:19.950819    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx" podUID="839984cb-3e33-4182-b323-751344047d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 19 11:55:20.542248 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:20.542218    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:20.907486 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:20.907444    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:55:20.949596 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:20.949545    2584 patch_prober.go:28] interesting pod/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 19 11:55:20.949804 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:20.949632    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx" podUID="839984cb-3e33-4182-b323-751344047d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 19 11:55:20.983615 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:20.983583    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:21.393389 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.393352    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:21.494023 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.493951    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:21.719040 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.718022    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"]
Sep 19 11:55:21.719040 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.718848    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="138969f5-591e-4ede-8d44-0f568c959000" containerName="istio-proxy"
Sep 19 11:55:21.719040 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.718869    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="138969f5-591e-4ede-8d44-0f568c959000" containerName="istio-proxy"
Sep 19 11:55:21.719335 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.719047    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="138969f5-591e-4ede-8d44-0f568c959000" containerName="istio-proxy"
Sep 19 11:55:21.722836 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.722800    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.725781 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.725751    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c7b650-openshift-default-dockercfg-whm5z\""
Sep 19 11:55:21.734662 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.734612    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"]
Sep 19 11:55:21.752437 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.752386    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:21.818962 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.818172    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-credential-socket\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.818962 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.818212    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d83f095e-42c9-4369-9e63-e29edc8ddd54-istiod-ca-cert\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.818962 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.818248    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh7mj\" (UniqueName: \"kubernetes.io/projected/d83f095e-42c9-4369-9e63-e29edc8ddd54-kube-api-access-qh7mj\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.818962 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.818282    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-workload-socket\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.818962 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.818347    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-workload-certs\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.818962 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.818425    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-data\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.818962 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.818501    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-token\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.818962 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.818534    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-podinfo\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.818962 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.818626    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-envoy\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.919675 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.919617    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-workload-socket\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.919675 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.919683    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-workload-certs\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.920191 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.919743    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-data\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.920191 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.919806    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-token\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.920191 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.919835    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-podinfo\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.920191 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.919890    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-envoy\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.920191 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.919957    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-credential-socket\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.920191 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.919980    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d83f095e-42c9-4369-9e63-e29edc8ddd54-istiod-ca-cert\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.920191 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.920034    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh7mj\" (UniqueName: \"kubernetes.io/projected/d83f095e-42c9-4369-9e63-e29edc8ddd54-kube-api-access-qh7mj\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.920191 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.920158    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-workload-certs\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.920569 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.920363    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-workload-socket\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.920569 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.920421    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-credential-socket\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.920569 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.920519    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-data\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.920929 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.920895    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d83f095e-42c9-4369-9e63-e29edc8ddd54-istiod-ca-cert\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.922581 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.922558    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-envoy\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.922796 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.922777    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-podinfo\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.928671 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.928639    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh7mj\" (UniqueName: \"kubernetes.io/projected/d83f095e-42c9-4369-9e63-e29edc8ddd54-kube-api-access-qh7mj\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.928869 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.928848    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-token\") pod \"e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") " pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:21.950185 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.950149    2584 patch_prober.go:28] interesting pod/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 19 11:55:21.950361 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.950222    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx" podUID="839984cb-3e33-4182-b323-751344047d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 19 11:55:21.966875 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:21.966841    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:22.038029 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:22.037927    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:22.135098 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:22.134879    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:22.335008 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:22.334980    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"]
Sep 19 11:55:22.336491 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:55:22.336459    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83f095e_42c9_4369_9e63_e29edc8ddd54.slice/crio-a423c9ee7d84a1bb70cba7d7e699b640c3fedde118c29b2124826b0b0752b574 WatchSource:0}: Error finding container a423c9ee7d84a1bb70cba7d7e699b640c3fedde118c29b2124826b0b0752b574: Status 404 returned error can't find the container with id a423c9ee7d84a1bb70cba7d7e699b640c3fedde118c29b2124826b0b0752b574
Sep 19 11:55:22.339407 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:22.339363    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:55:22.339500 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:22.339465    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:55:22.339553 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:22.339512    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:55:22.949381 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:22.949347    2584 patch_prober.go:28] interesting pod/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 19 11:55:22.949791 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:22.949421    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx" podUID="839984cb-3e33-4182-b323-751344047d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 19 11:55:23.111882 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:23.111840    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf" event={"ID":"d83f095e-42c9-4369-9e63-e29edc8ddd54","Type":"ContainerStarted","Data":"c5573872b1c92b0be474c49c50b667b8586533334cf356266b9b3ac0db205abf"}
Sep 19 11:55:23.112238 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:23.112191    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf" event={"ID":"d83f095e-42c9-4369-9e63-e29edc8ddd54","Type":"ContainerStarted","Data":"a423c9ee7d84a1bb70cba7d7e699b640c3fedde118c29b2124826b0b0752b574"}
Sep 19 11:55:23.132526 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:23.132463    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf" podStartSLOduration=2.132444748 podStartE2EDuration="2.132444748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:55:23.131204603 +0000 UTC m=+1307.331234407" watchObservedRunningTime="2026-09-19 11:55:23.132444748 +0000 UTC m=+1307.332474550"
Sep 19 11:55:23.950096 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:23.950058    2584 patch_prober.go:28] interesting pod/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 19 11:55:23.950516 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:23.950115    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx" podUID="839984cb-3e33-4182-b323-751344047d86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 19 11:55:24.038749 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:24.038712    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:24.045029 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:24.044993    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:24.118520 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:24.118477    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:24.119557 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:24.119529    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:55:24.439552 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:24.439521    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:24.954155 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:24.954119    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:55:24.955337 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:24.955314    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:55:25.026350 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:25.026311    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:25.123868 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:25.123833    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:25.224556 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:25.224468    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:26.056257 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:26.056213    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:26.807065 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:26.807009    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:27.447497 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:27.447464    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:28.151425 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:28.151380    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:28.774036 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:28.774001    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:29.566432 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:29.566400    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:29.618756 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:29.618710    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:30.001804 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:30.001715    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:30.104580 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:30.104545    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:30.163396 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:30.163365    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:51.349615 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:51.349534    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:51.547609 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:51.547573    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:52.071378 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:52.071339    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:52.772489 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:52.772457    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:53.664341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:53.664299    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:54.286571 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:54.286541    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:55.047510 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:55.047478    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:55.148053 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:55.148019    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:55.695877 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:55.695844    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:56.438039 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:56.438000    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:58.262106 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:58.262048    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:59.065074 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:59.065048    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:55:59.854984 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:55:59.854953    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:00.589405 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:00.589347    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:01.271561 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:01.271453    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:01.964955 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:01.964883    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:02.772692 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:02.772662    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:03.395301 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:03.395263    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:04.045821 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:04.045792    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:05.437335 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:05.437300    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:08.152523 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:08.152481    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:09.119813 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:09.119774    2584 patch_prober.go:28] interesting pod/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 19 11:56:09.120095 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:09.119844    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf" podUID="d83f095e-42c9-4369-9e63-e29edc8ddd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 19 11:56:09.388942 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:09.388815    2584 generic.go:356] "Generic (PLEG): container finished" podID="d83f095e-42c9-4369-9e63-e29edc8ddd54" containerID="c5573872b1c92b0be474c49c50b667b8586533334cf356266b9b3ac0db205abf" exitCode=0
Sep 19 11:56:09.388942 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:09.388893    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf" event={"ID":"d83f095e-42c9-4369-9e63-e29edc8ddd54","Type":"ContainerDied","Data":"c5573872b1c92b0be474c49c50b667b8586533334cf356266b9b3ac0db205abf"}
Sep 19 11:56:09.389484 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:09.389466    2584 scope.go:117] "RemoveContainer" containerID="c5573872b1c92b0be474c49c50b667b8586533334cf356266b9b3ac0db205abf"
Sep 19 11:56:09.391094 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:09.391055    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:56:09.391217 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:09.391148    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:56:09.391217 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:09.391197    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 11:56:10.399392 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:10.399355    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf" event={"ID":"d83f095e-42c9-4369-9e63-e29edc8ddd54","Type":"ContainerStarted","Data":"edf27edd1cc4830f878ffbb5d9ea1531f4c2b782686229715ff3ca11a28335aa"}
Sep 19 11:56:11.038923 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:11.038882    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:56:11.039113 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:11.039079    2584 patch_prober.go:28] interesting pod/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 19 11:56:11.039159 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:11.039138    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf" podUID="d83f095e-42c9-4369-9e63-e29edc8ddd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 19 11:56:12.038820 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:12.038781    2584 patch_prober.go:28] interesting pod/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 19 11:56:12.039272 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:12.038835    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf" podUID="d83f095e-42c9-4369-9e63-e29edc8ddd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 19 11:56:13.038567 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:13.038526    2584 patch_prober.go:28] interesting pod/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 19 11:56:13.038764 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:13.038590    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf" podUID="d83f095e-42c9-4369-9e63-e29edc8ddd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 19 11:56:13.714368 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:13.713660    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:13.779748 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:13.779715    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:13.876806 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:13.876775    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:13.933531 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:13.933496    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:14.039283 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:14.039190    2584 patch_prober.go:28] interesting pod/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 19 11:56:14.039283 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:14.039248    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf" podUID="d83f095e-42c9-4369-9e63-e29edc8ddd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 19 11:56:14.258671 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:14.258599    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:14.367588 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:14.367559    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:15.038833 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:15.038798    2584 patch_prober.go:28] interesting pod/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 19 11:56:15.039379 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:15.038885    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf" podUID="d83f095e-42c9-4369-9e63-e29edc8ddd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 19 11:56:15.054826 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:15.054793    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:15.949354 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:15.949322    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:16.038649 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:16.038606    2584 patch_prober.go:28] interesting pod/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 19 11:56:16.038810 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:16.038670    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf" podUID="d83f095e-42c9-4369-9e63-e29edc8ddd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 19 11:56:16.684461 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:16.684416    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:17.043174 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:17.043084    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:56:17.043414 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:17.043395    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:56:17.044194 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:17.044173    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:56:17.356026 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:17.355993    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:18.127334 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:18.127304    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:18.850719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:18.850686    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:20.048438 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:20.048390    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:20.106390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:20.106348    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:20.284284 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:20.284247    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"]
Sep 19 11:56:20.284532 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:20.284506    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx" podUID="839984cb-3e33-4182-b323-751344047d86" containerName="istio-proxy" containerID="cri-o://85b24fce01ee96b7bbe22547f563c6102fe1a737c6032ac4b0819af713fe97e7" gracePeriod=30
Sep 19 11:56:20.300654 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:20.300563    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:20.959521 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:20.959491    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:21.917000 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:21.916956    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:22.548032 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:22.547987    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:23.431118 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:23.431083    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:24.079828 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:24.079796    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:24.752671 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:24.752631    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:24.956494 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:24.956459    2584 patch_prober.go:28] interesting pod/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 19 11:56:24.956689 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:24.956522    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx" podUID="839984cb-3e33-4182-b323-751344047d86" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 11:56:25.467316 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.467290    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:25.495338 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.495300    2584 generic.go:356] "Generic (PLEG): container finished" podID="839984cb-3e33-4182-b323-751344047d86" containerID="85b24fce01ee96b7bbe22547f563c6102fe1a737c6032ac4b0819af713fe97e7" exitCode=0
Sep 19 11:56:25.495338 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.495323    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx" event={"ID":"839984cb-3e33-4182-b323-751344047d86","Type":"ContainerDied","Data":"85b24fce01ee96b7bbe22547f563c6102fe1a737c6032ac4b0819af713fe97e7"}
Sep 19 11:56:25.495597 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.495395    2584 scope.go:117] "RemoveContainer" containerID="cb0a35960a13be190c1b439931a9de2b249c98d6bf8eebcd4dda8c008b881cb6"
Sep 19 11:56:25.554299 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.554222    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:56:25.710396 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.710363    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839984cb-3e33-4182-b323-751344047d86-istiod-ca-cert\") pod \"839984cb-3e33-4182-b323-751344047d86\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") "
Sep 19 11:56:25.710615 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.710426    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-workload-certs\") pod \"839984cb-3e33-4182-b323-751344047d86\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") "
Sep 19 11:56:25.710615 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.710565    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/839984cb-3e33-4182-b323-751344047d86-istio-token\") pod \"839984cb-3e33-4182-b323-751344047d86\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") "
Sep 19 11:56:25.710745 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.710617    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-istio-data\") pod \"839984cb-3e33-4182-b323-751344047d86\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") "
Sep 19 11:56:25.710745 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.710728    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839984cb-3e33-4182-b323-751344047d86-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "839984cb-3e33-4182-b323-751344047d86" (UID: "839984cb-3e33-4182-b323-751344047d86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:56:25.710860 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.710739    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-credential-socket\") pod \"839984cb-3e33-4182-b323-751344047d86\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") "
Sep 19 11:56:25.710860 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.710761    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "839984cb-3e33-4182-b323-751344047d86" (UID: "839984cb-3e33-4182-b323-751344047d86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:56:25.710860 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.710785    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wghl6\" (UniqueName: \"kubernetes.io/projected/839984cb-3e33-4182-b323-751344047d86-kube-api-access-wghl6\") pod \"839984cb-3e33-4182-b323-751344047d86\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") "
Sep 19 11:56:25.710860 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.710841    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "839984cb-3e33-4182-b323-751344047d86" (UID: "839984cb-3e33-4182-b323-751344047d86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:56:25.710860 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.710855    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-workload-socket\") pod \"839984cb-3e33-4182-b323-751344047d86\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") "
Sep 19 11:56:25.711147 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.710870    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-istio-data" (OuterVolumeSpecName: "istio-data") pod "839984cb-3e33-4182-b323-751344047d86" (UID: "839984cb-3e33-4182-b323-751344047d86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:56:25.711147 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.710969    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-istio-envoy\") pod \"839984cb-3e33-4182-b323-751344047d86\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") "
Sep 19 11:56:25.711147 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.711003    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839984cb-3e33-4182-b323-751344047d86-istio-podinfo\") pod \"839984cb-3e33-4182-b323-751344047d86\" (UID: \"839984cb-3e33-4182-b323-751344047d86\") "
Sep 19 11:56:25.711147 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.711105    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "839984cb-3e33-4182-b323-751344047d86" (UID: "839984cb-3e33-4182-b323-751344047d86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:56:25.711401 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.711384    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839984cb-3e33-4182-b323-751344047d86-istiod-ca-cert\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:25.711466 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.711406    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-workload-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:25.711466 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.711419    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-istio-data\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:25.711466 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.711432    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-credential-socket\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:25.711466 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.711447    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-workload-socket\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:25.713059 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.713033    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839984cb-3e33-4182-b323-751344047d86-kube-api-access-wghl6" (OuterVolumeSpecName: "kube-api-access-wghl6") pod "839984cb-3e33-4182-b323-751344047d86" (UID: "839984cb-3e33-4182-b323-751344047d86"). InnerVolumeSpecName "kube-api-access-wghl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:56:25.713214 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.713196    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/839984cb-3e33-4182-b323-751344047d86-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "839984cb-3e33-4182-b323-751344047d86" (UID: "839984cb-3e33-4182-b323-751344047d86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 11:56:25.713523 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.713500    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839984cb-3e33-4182-b323-751344047d86-istio-token" (OuterVolumeSpecName: "istio-token") pod "839984cb-3e33-4182-b323-751344047d86" (UID: "839984cb-3e33-4182-b323-751344047d86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:56:25.713523 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.713505    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "839984cb-3e33-4182-b323-751344047d86" (UID: "839984cb-3e33-4182-b323-751344047d86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:56:25.812029 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.812001    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/839984cb-3e33-4182-b323-751344047d86-istio-token\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:25.812029 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.812030    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wghl6\" (UniqueName: \"kubernetes.io/projected/839984cb-3e33-4182-b323-751344047d86-kube-api-access-wghl6\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:25.812442 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.812040    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839984cb-3e33-4182-b323-751344047d86-istio-envoy\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:25.812442 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:25.812049    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839984cb-3e33-4182-b323-751344047d86-istio-podinfo\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:26.078092 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:26.078012    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:26.503826 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:26.503732    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"
Sep 19 11:56:26.503826 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:26.503767    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx" event={"ID":"839984cb-3e33-4182-b323-751344047d86","Type":"ContainerDied","Data":"e33495c78093749228efd42243e803f501051c3b5fa355d954ebfc5102723de0"}
Sep 19 11:56:26.504112 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:26.503833    2584 scope.go:117] "RemoveContainer" containerID="85b24fce01ee96b7bbe22547f563c6102fe1a737c6032ac4b0819af713fe97e7"
Sep 19 11:56:26.523453 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:26.523422    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"]
Sep 19 11:56:26.528749 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:26.528722    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3c21ca-openshift-default-586f6db9f8-hkhzx"]
Sep 19 11:56:27.906112 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:27.906070    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="839984cb-3e33-4182-b323-751344047d86" path="/var/lib/kubelet/pods/839984cb-3e33-4182-b323-751344047d86/volumes"
Sep 19 11:56:36.095226 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:36.095186    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:36.157514 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:36.157480    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:36.212021 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:36.211989    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:36.279029 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:36.278991    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:36.332849 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:36.332818    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:36.627682 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:36.627649    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:36.725112 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:36.725067    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:37.462623 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:37.462578    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:38.258891 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:38.258857    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:39.026318 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:39.026284    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:39.820417 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:39.820378    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:40.459045 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:40.459015    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:42.298518 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:42.298487    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:42.399250 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:42.399208    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:42.526456 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:42.526425    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"]
Sep 19 11:56:42.526700 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:42.526675    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf" podUID="d83f095e-42c9-4369-9e63-e29edc8ddd54" containerName="istio-proxy" containerID="cri-o://edf27edd1cc4830f878ffbb5d9ea1531f4c2b782686229715ff3ca11a28335aa" gracePeriod=30
Sep 19 11:56:42.540377 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:42.540350    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:43.147685 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:43.147647    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:43.880645 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:43.880607    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:44.676184 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:44.676145    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:45.357758 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:45.357728    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:46.119420 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:46.119392    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:46.804901 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:46.804854    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:47.046645 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.046600    2584 patch_prober.go:28] interesting pod/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 19 11:56:47.046855 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.046690    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf" podUID="d83f095e-42c9-4369-9e63-e29edc8ddd54" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 11:56:47.438211 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.438183    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:47.802173 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.802145    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:56:47.931138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.931088    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-workload-socket\") pod \"d83f095e-42c9-4369-9e63-e29edc8ddd54\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") "
Sep 19 11:56:47.931138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.931153    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-token\") pod \"d83f095e-42c9-4369-9e63-e29edc8ddd54\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") "
Sep 19 11:56:47.931719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.931187    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-envoy\") pod \"d83f095e-42c9-4369-9e63-e29edc8ddd54\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") "
Sep 19 11:56:47.931719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.931220    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh7mj\" (UniqueName: \"kubernetes.io/projected/d83f095e-42c9-4369-9e63-e29edc8ddd54-kube-api-access-qh7mj\") pod \"d83f095e-42c9-4369-9e63-e29edc8ddd54\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") "
Sep 19 11:56:47.931719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.931245    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-workload-certs\") pod \"d83f095e-42c9-4369-9e63-e29edc8ddd54\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") "
Sep 19 11:56:47.931719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.931320    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-credential-socket\") pod \"d83f095e-42c9-4369-9e63-e29edc8ddd54\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") "
Sep 19 11:56:47.931719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.931364    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-podinfo\") pod \"d83f095e-42c9-4369-9e63-e29edc8ddd54\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") "
Sep 19 11:56:47.931719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.931402    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-data\") pod \"d83f095e-42c9-4369-9e63-e29edc8ddd54\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") "
Sep 19 11:56:47.931719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.931400    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d83f095e-42c9-4369-9e63-e29edc8ddd54" (UID: "d83f095e-42c9-4369-9e63-e29edc8ddd54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:56:47.931719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.931443    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d83f095e-42c9-4369-9e63-e29edc8ddd54-istiod-ca-cert\") pod \"d83f095e-42c9-4369-9e63-e29edc8ddd54\" (UID: \"d83f095e-42c9-4369-9e63-e29edc8ddd54\") "
Sep 19 11:56:47.931719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.931608    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d83f095e-42c9-4369-9e63-e29edc8ddd54" (UID: "d83f095e-42c9-4369-9e63-e29edc8ddd54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:56:47.931719 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.931638    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d83f095e-42c9-4369-9e63-e29edc8ddd54" (UID: "d83f095e-42c9-4369-9e63-e29edc8ddd54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:56:47.932458 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.931798    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-workload-socket\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:47.932458 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.931819    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-workload-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:47.932458 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.931832    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-credential-socket\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:47.932458 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.931829    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-data" (OuterVolumeSpecName: "istio-data") pod "d83f095e-42c9-4369-9e63-e29edc8ddd54" (UID: "d83f095e-42c9-4369-9e63-e29edc8ddd54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:56:47.932458 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.931903    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d83f095e-42c9-4369-9e63-e29edc8ddd54-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d83f095e-42c9-4369-9e63-e29edc8ddd54" (UID: "d83f095e-42c9-4369-9e63-e29edc8ddd54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:56:47.933698 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.933665    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d83f095e-42c9-4369-9e63-e29edc8ddd54" (UID: "d83f095e-42c9-4369-9e63-e29edc8ddd54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 11:56:47.933818 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.933744    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-token" (OuterVolumeSpecName: "istio-token") pod "d83f095e-42c9-4369-9e63-e29edc8ddd54" (UID: "d83f095e-42c9-4369-9e63-e29edc8ddd54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:56:47.934067 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.934045    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d83f095e-42c9-4369-9e63-e29edc8ddd54" (UID: "d83f095e-42c9-4369-9e63-e29edc8ddd54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:56:47.934187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:47.934045    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d83f095e-42c9-4369-9e63-e29edc8ddd54-kube-api-access-qh7mj" (OuterVolumeSpecName: "kube-api-access-qh7mj") pod "d83f095e-42c9-4369-9e63-e29edc8ddd54" (UID: "d83f095e-42c9-4369-9e63-e29edc8ddd54"). InnerVolumeSpecName "kube-api-access-qh7mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:56:48.033184 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:48.033072    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d83f095e-42c9-4369-9e63-e29edc8ddd54-istiod-ca-cert\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:48.033184 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:48.033113    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-token\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:48.033184 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:48.033129    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-envoy\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:48.033184 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:48.033143    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh7mj\" (UniqueName: \"kubernetes.io/projected/d83f095e-42c9-4369-9e63-e29edc8ddd54-kube-api-access-qh7mj\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:48.033184 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:48.033158    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-podinfo\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:48.033184 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:48.033171    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d83f095e-42c9-4369-9e63-e29edc8ddd54-istio-data\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:48.631099 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:48.631060    2584 generic.go:356] "Generic (PLEG): container finished" podID="d83f095e-42c9-4369-9e63-e29edc8ddd54" containerID="edf27edd1cc4830f878ffbb5d9ea1531f4c2b782686229715ff3ca11a28335aa" exitCode=0
Sep 19 11:56:48.631254 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:48.631130    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"
Sep 19 11:56:48.631254 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:48.631126    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf" event={"ID":"d83f095e-42c9-4369-9e63-e29edc8ddd54","Type":"ContainerDied","Data":"edf27edd1cc4830f878ffbb5d9ea1531f4c2b782686229715ff3ca11a28335aa"}
Sep 19 11:56:48.631355 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:48.631280    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf" event={"ID":"d83f095e-42c9-4369-9e63-e29edc8ddd54","Type":"ContainerDied","Data":"a423c9ee7d84a1bb70cba7d7e699b640c3fedde118c29b2124826b0b0752b574"}
Sep 19 11:56:48.631355 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:48.631311    2584 scope.go:117] "RemoveContainer" containerID="edf27edd1cc4830f878ffbb5d9ea1531f4c2b782686229715ff3ca11a28335aa"
Sep 19 11:56:48.641654 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:48.641631    2584 scope.go:117] "RemoveContainer" containerID="c5573872b1c92b0be474c49c50b667b8586533334cf356266b9b3ac0db205abf"
Sep 19 11:56:48.651101 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:48.650793    2584 scope.go:117] "RemoveContainer" containerID="edf27edd1cc4830f878ffbb5d9ea1531f4c2b782686229715ff3ca11a28335aa"
Sep 19 11:56:48.651527 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:56:48.651492    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edf27edd1cc4830f878ffbb5d9ea1531f4c2b782686229715ff3ca11a28335aa\": container with ID starting with edf27edd1cc4830f878ffbb5d9ea1531f4c2b782686229715ff3ca11a28335aa not found: ID does not exist" containerID="edf27edd1cc4830f878ffbb5d9ea1531f4c2b782686229715ff3ca11a28335aa"
Sep 19 11:56:48.651631 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:48.651540    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edf27edd1cc4830f878ffbb5d9ea1531f4c2b782686229715ff3ca11a28335aa"} err="failed to get container status \"edf27edd1cc4830f878ffbb5d9ea1531f4c2b782686229715ff3ca11a28335aa\": rpc error: code = NotFound desc = could not find container \"edf27edd1cc4830f878ffbb5d9ea1531f4c2b782686229715ff3ca11a28335aa\": container with ID starting with edf27edd1cc4830f878ffbb5d9ea1531f4c2b782686229715ff3ca11a28335aa not found: ID does not exist"
Sep 19 11:56:48.651631 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:48.651565    2584 scope.go:117] "RemoveContainer" containerID="c5573872b1c92b0be474c49c50b667b8586533334cf356266b9b3ac0db205abf"
Sep 19 11:56:48.651901 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:56:48.651878    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5573872b1c92b0be474c49c50b667b8586533334cf356266b9b3ac0db205abf\": container with ID starting with c5573872b1c92b0be474c49c50b667b8586533334cf356266b9b3ac0db205abf not found: ID does not exist" containerID="c5573872b1c92b0be474c49c50b667b8586533334cf356266b9b3ac0db205abf"
Sep 19 11:56:48.651995 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:48.651938    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5573872b1c92b0be474c49c50b667b8586533334cf356266b9b3ac0db205abf"} err="failed to get container status \"c5573872b1c92b0be474c49c50b667b8586533334cf356266b9b3ac0db205abf\": rpc error: code = NotFound desc = could not find container \"c5573872b1c92b0be474c49c50b667b8586533334cf356266b9b3ac0db205abf\": container with ID starting with c5573872b1c92b0be474c49c50b667b8586533334cf356266b9b3ac0db205abf not found: ID does not exist"
Sep 19 11:56:48.653679 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:48.653658    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"]
Sep 19 11:56:48.656930 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:48.656889    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c7b650-openshift-default-6db4547656-t8sbf"]
Sep 19 11:56:49.904463 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:49.904424    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d83f095e-42c9-4369-9e63-e29edc8ddd54" path="/var/lib/kubelet/pods/d83f095e-42c9-4369-9e63-e29edc8ddd54/volumes"
Sep 19 11:56:53.566036 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:53.566001    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"]
Sep 19 11:56:53.566506 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:53.566221    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf" podUID="2854ecea-0ccb-4ddc-ac6d-1f8a601d2789" containerName="maas-api" containerID="cri-o://1bd2156abf1920c73e3aec28d663c9f1a1e3f4fafa8230553d2589592714516b" gracePeriod=30
Sep 19 11:56:53.589121 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:53.588749    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:53.684889 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:53.684860    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:53.763305 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:53.763266    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:53.825818 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:53.825728    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:53.828746 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:53.828724    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"
Sep 19 11:56:53.894955 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:53.894898    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-maas-api-tls\") pod \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\" (UID: \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\") "
Sep 19 11:56:53.895138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:53.894979    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-metrics-certs\") pod \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\" (UID: \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\") "
Sep 19 11:56:53.895138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:53.895020    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdp85\" (UniqueName: \"kubernetes.io/projected/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-kube-api-access-sdp85\") pod \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\" (UID: \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\") "
Sep 19 11:56:53.895138 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:53.895077    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-openshift-service-ca\") pod \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\" (UID: \"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789\") "
Sep 19 11:56:53.895536 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:53.895509    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2854ecea-0ccb-4ddc-ac6d-1f8a601d2789" (UID: "2854ecea-0ccb-4ddc-ac6d-1f8a601d2789"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:56:53.897386 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:53.897351    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-kube-api-access-sdp85" (OuterVolumeSpecName: "kube-api-access-sdp85") pod "2854ecea-0ccb-4ddc-ac6d-1f8a601d2789" (UID: "2854ecea-0ccb-4ddc-ac6d-1f8a601d2789"). InnerVolumeSpecName "kube-api-access-sdp85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:56:53.897501 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:53.897412    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2854ecea-0ccb-4ddc-ac6d-1f8a601d2789" (UID: "2854ecea-0ccb-4ddc-ac6d-1f8a601d2789"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:56:53.897642 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:53.897600    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2854ecea-0ccb-4ddc-ac6d-1f8a601d2789" (UID: "2854ecea-0ccb-4ddc-ac6d-1f8a601d2789"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:56:53.997159 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:53.997121    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-maas-api-tls\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:53.997159 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:53.997157    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-metrics-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:53.997159 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:53.997171    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdp85\" (UniqueName: \"kubernetes.io/projected/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-kube-api-access-sdp85\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:53.997447 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:53.997184    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789-openshift-service-ca\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:56:54.120687 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:54.120600    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:54.246453 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:54.246417    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:54.670591 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:54.670557    2584 generic.go:356] "Generic (PLEG): container finished" podID="2854ecea-0ccb-4ddc-ac6d-1f8a601d2789" containerID="1bd2156abf1920c73e3aec28d663c9f1a1e3f4fafa8230553d2589592714516b" exitCode=0
Sep 19 11:56:54.671116 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:54.670613    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"
Sep 19 11:56:54.671116 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:54.670650    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf" event={"ID":"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789","Type":"ContainerDied","Data":"1bd2156abf1920c73e3aec28d663c9f1a1e3f4fafa8230553d2589592714516b"}
Sep 19 11:56:54.671116 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:54.670706    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf" event={"ID":"2854ecea-0ccb-4ddc-ac6d-1f8a601d2789","Type":"ContainerDied","Data":"aa47b49c6212816d5d8180affc3ab8d274a112c2b52a0d04a5a36b872bf92158"}
Sep 19 11:56:54.671116 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:54.670730    2584 scope.go:117] "RemoveContainer" containerID="1bd2156abf1920c73e3aec28d663c9f1a1e3f4fafa8230553d2589592714516b"
Sep 19 11:56:54.680716 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:54.680688    2584 scope.go:117] "RemoveContainer" containerID="1bd2156abf1920c73e3aec28d663c9f1a1e3f4fafa8230553d2589592714516b"
Sep 19 11:56:54.681050 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:56:54.681019    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bd2156abf1920c73e3aec28d663c9f1a1e3f4fafa8230553d2589592714516b\": container with ID starting with 1bd2156abf1920c73e3aec28d663c9f1a1e3f4fafa8230553d2589592714516b not found: ID does not exist" containerID="1bd2156abf1920c73e3aec28d663c9f1a1e3f4fafa8230553d2589592714516b"
Sep 19 11:56:54.681155 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:54.681058    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd2156abf1920c73e3aec28d663c9f1a1e3f4fafa8230553d2589592714516b"} err="failed to get container status \"1bd2156abf1920c73e3aec28d663c9f1a1e3f4fafa8230553d2589592714516b\": rpc error: code = NotFound desc = could not find container \"1bd2156abf1920c73e3aec28d663c9f1a1e3f4fafa8230553d2589592714516b\": container with ID starting with 1bd2156abf1920c73e3aec28d663c9f1a1e3f4fafa8230553d2589592714516b not found: ID does not exist"
Sep 19 11:56:54.687200 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:54.687174    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"]
Sep 19 11:56:54.690664 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:54.690641    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bb32d-5f74d7bfc6-5d5nf"]
Sep 19 11:56:54.891232 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:54.891198    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:55.764450 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:55.764415    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:55.903535 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:55.903501    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2854ecea-0ccb-4ddc-ac6d-1f8a601d2789" path="/var/lib/kubelet/pods/2854ecea-0ccb-4ddc-ac6d-1f8a601d2789/volumes"
Sep 19 11:56:56.528565 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:56.528532    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:57.146184 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:57.146151    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:58.566281 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:58.566246    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:59.886766 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:59.886735    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:56:59.940954 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:56:59.940895    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:00.142468 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:00.142379    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:00.640448 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:00.640420    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:01.348756 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:01.348721    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:02.148838 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:02.148802    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:02.789219 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:02.789185    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:03.449339 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:03.449309    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:04.077876 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:04.077829    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:07.900480 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:07.900438    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:08.355821 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:08.355770    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:09.151423 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:09.151365    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:09.780282 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:09.780241    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:10.451535 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:10.451495    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:11.263118 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:11.263042    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:11.943730 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:11.943697    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:12.488612 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:12.488570    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:12.553104 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:12.553058    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:12.613270 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:12.613240    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:12.667115 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:12.667081    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:12.723259 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:12.723224    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:12.891438 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:12.891393    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"]
Sep 19 11:57:12.910854 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:12.910815    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"]
Sep 19 11:57:12.923498 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:12.923464    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"]
Sep 19 11:57:13.030402 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:13.030374    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:13.090535 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:13.090488    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:13.149955 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:13.149836    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:13.805723 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:13.805689    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:14.458504 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:14.458467    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:15.258949 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:15.258894    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:18.165668 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:18.165622    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:19.022878 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:19.022847    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:19.122950 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:19.122894    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:19.261543 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:19.261496    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"]
Sep 19 11:57:19.262242 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:19.262018    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn" podUID="81bbf18e-1913-4789-a8b0-e8bf23d0b99a" containerName="istio-proxy" containerID="cri-o://a915d714ab2d957c45ec795bc46a53ed0cc46649d4805ca3bc2bf5fd38fdadd5" gracePeriod=30
Sep 19 11:57:19.271726 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:19.271704    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:22.827245 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:22.827211    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 19 11:57:22.827658 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:22.827276    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn" podUID="81bbf18e-1913-4789-a8b0-e8bf23d0b99a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 11:57:23.724635 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:23.724590    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"]
Sep 19 11:57:23.724931 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:23.724885    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl" podUID="64ffcacc-383a-4bf4-bb7e-1e03ebafdf65" containerName="maas-api" containerID="cri-o://26bc0f6c3d638227dd5217d01bea399a763d12d93cbb978441367afa29d484e7" gracePeriod=30
Sep 19 11:57:23.751780 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:23.751757    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:23.812124 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:23.812053    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:23.872601 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:23.872567    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:23.981171 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:23.981087    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:23.986633 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:23.986609    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"
Sep 19 11:57:24.098744 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.098702    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-maas-api-tls\") pod \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\" (UID: \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\") "
Sep 19 11:57:24.098906 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.098849    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-metrics-certs\") pod \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\" (UID: \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\") "
Sep 19 11:57:24.098996 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.098904    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-openshift-service-ca\") pod \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\" (UID: \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\") "
Sep 19 11:57:24.098996 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.098978    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssmvz\" (UniqueName: \"kubernetes.io/projected/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-kube-api-access-ssmvz\") pod \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\" (UID: \"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65\") "
Sep 19 11:57:24.099372 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.099335    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "64ffcacc-383a-4bf4-bb7e-1e03ebafdf65" (UID: "64ffcacc-383a-4bf4-bb7e-1e03ebafdf65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:57:24.101810 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.101762    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-kube-api-access-ssmvz" (OuterVolumeSpecName: "kube-api-access-ssmvz") pod "64ffcacc-383a-4bf4-bb7e-1e03ebafdf65" (UID: "64ffcacc-383a-4bf4-bb7e-1e03ebafdf65"). InnerVolumeSpecName "kube-api-access-ssmvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:57:24.101941 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.101861    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "64ffcacc-383a-4bf4-bb7e-1e03ebafdf65" (UID: "64ffcacc-383a-4bf4-bb7e-1e03ebafdf65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:57:24.101941 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.101888    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "64ffcacc-383a-4bf4-bb7e-1e03ebafdf65" (UID: "64ffcacc-383a-4bf4-bb7e-1e03ebafdf65"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:57:24.200687 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.200643    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-metrics-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:24.200687 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.200679    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-openshift-service-ca\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:24.200687 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.200693    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssmvz\" (UniqueName: \"kubernetes.io/projected/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-kube-api-access-ssmvz\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:24.200971 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.200705    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65-maas-api-tls\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:24.298709 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.298658    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:24.444244 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.444167    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:24.509265 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.509239    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:57:24.604823 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.604787    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmfjh\" (UniqueName: \"kubernetes.io/projected/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-kube-api-access-xmfjh\") pod \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") "
Sep 19 11:57:24.604823 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.604834    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-workload-certs\") pod \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") "
Sep 19 11:57:24.605144 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.604885    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-workload-socket\") pod \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") "
Sep 19 11:57:24.605144 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.604970    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istiod-ca-cert\") pod \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") "
Sep 19 11:57:24.605144 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.604987    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-token\") pod \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") "
Sep 19 11:57:24.605144 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.605020    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-credential-socket\") pod \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") "
Sep 19 11:57:24.605144 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.605041    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-envoy\") pod \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") "
Sep 19 11:57:24.605144 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.605081    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-podinfo\") pod \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") "
Sep 19 11:57:24.605144 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.605113    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-data\") pod \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\" (UID: \"81bbf18e-1913-4789-a8b0-e8bf23d0b99a\") "
Sep 19 11:57:24.605517 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.605272    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "81bbf18e-1913-4789-a8b0-e8bf23d0b99a" (UID: "81bbf18e-1913-4789-a8b0-e8bf23d0b99a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:57:24.605517 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.605259    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "81bbf18e-1913-4789-a8b0-e8bf23d0b99a" (UID: "81bbf18e-1913-4789-a8b0-e8bf23d0b99a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:57:24.605517 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.605393    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "81bbf18e-1913-4789-a8b0-e8bf23d0b99a" (UID: "81bbf18e-1913-4789-a8b0-e8bf23d0b99a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 11:57:24.605517 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.605491    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "81bbf18e-1913-4789-a8b0-e8bf23d0b99a" (UID: "81bbf18e-1913-4789-a8b0-e8bf23d0b99a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:57:24.605845 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.605591    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-credential-socket\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:24.605845 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.605624    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-workload-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:24.605845 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.605639    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-workload-socket\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:24.605845 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.605651    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istiod-ca-cert\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:24.605845 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.605784    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-data" (OuterVolumeSpecName: "istio-data") pod "81bbf18e-1913-4789-a8b0-e8bf23d0b99a" (UID: "81bbf18e-1913-4789-a8b0-e8bf23d0b99a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:57:24.607487 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.607455    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-token" (OuterVolumeSpecName: "istio-token") pod "81bbf18e-1913-4789-a8b0-e8bf23d0b99a" (UID: "81bbf18e-1913-4789-a8b0-e8bf23d0b99a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:57:24.607487 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.607472    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-kube-api-access-xmfjh" (OuterVolumeSpecName: "kube-api-access-xmfjh") pod "81bbf18e-1913-4789-a8b0-e8bf23d0b99a" (UID: "81bbf18e-1913-4789-a8b0-e8bf23d0b99a"). InnerVolumeSpecName "kube-api-access-xmfjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:57:24.607864 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.607846    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "81bbf18e-1913-4789-a8b0-e8bf23d0b99a" (UID: "81bbf18e-1913-4789-a8b0-e8bf23d0b99a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:57:24.607906 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.607876    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "81bbf18e-1913-4789-a8b0-e8bf23d0b99a" (UID: "81bbf18e-1913-4789-a8b0-e8bf23d0b99a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 11:57:24.706473 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.706427    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-envoy\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:24.706473 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.706465    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-podinfo\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:24.706473 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.706474    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-data\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:24.706727 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.706486    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmfjh\" (UniqueName: \"kubernetes.io/projected/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-kube-api-access-xmfjh\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:24.706727 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.706496    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81bbf18e-1913-4789-a8b0-e8bf23d0b99a-istio-token\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:24.840192 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.840155    2584 generic.go:356] "Generic (PLEG): container finished" podID="81bbf18e-1913-4789-a8b0-e8bf23d0b99a" containerID="a915d714ab2d957c45ec795bc46a53ed0cc46649d4805ca3bc2bf5fd38fdadd5" exitCode=0
Sep 19 11:57:24.840393 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.840223    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"
Sep 19 11:57:24.840393 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.840247    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn" event={"ID":"81bbf18e-1913-4789-a8b0-e8bf23d0b99a","Type":"ContainerDied","Data":"a915d714ab2d957c45ec795bc46a53ed0cc46649d4805ca3bc2bf5fd38fdadd5"}
Sep 19 11:57:24.840393 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.840350    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn" event={"ID":"81bbf18e-1913-4789-a8b0-e8bf23d0b99a","Type":"ContainerDied","Data":"429071f716b993627a4f367f7f79cde7190db2950ef835705af9eb8ee501df40"}
Sep 19 11:57:24.840393 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.840379    2584 scope.go:117] "RemoveContainer" containerID="a915d714ab2d957c45ec795bc46a53ed0cc46649d4805ca3bc2bf5fd38fdadd5"
Sep 19 11:57:24.841867 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.841843    2584 generic.go:356] "Generic (PLEG): container finished" podID="64ffcacc-383a-4bf4-bb7e-1e03ebafdf65" containerID="26bc0f6c3d638227dd5217d01bea399a763d12d93cbb978441367afa29d484e7" exitCode=0
Sep 19 11:57:24.841975 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.841939    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl" event={"ID":"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65","Type":"ContainerDied","Data":"26bc0f6c3d638227dd5217d01bea399a763d12d93cbb978441367afa29d484e7"}
Sep 19 11:57:24.841975 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.841905    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"
Sep 19 11:57:24.842099 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.841998    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl" event={"ID":"64ffcacc-383a-4bf4-bb7e-1e03ebafdf65","Type":"ContainerDied","Data":"95924e938865fa3b6f57a50e3f501e78b98bda22f3f34725c3620b0483f1d7f9"}
Sep 19 11:57:24.850524 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.850497    2584 scope.go:117] "RemoveContainer" containerID="7db13862736d672e47f320be121ddd06ffa4d5cca24611981e08df799d827ace"
Sep 19 11:57:24.860499 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.860478    2584 scope.go:117] "RemoveContainer" containerID="a915d714ab2d957c45ec795bc46a53ed0cc46649d4805ca3bc2bf5fd38fdadd5"
Sep 19 11:57:24.860851 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:57:24.860829    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a915d714ab2d957c45ec795bc46a53ed0cc46649d4805ca3bc2bf5fd38fdadd5\": container with ID starting with a915d714ab2d957c45ec795bc46a53ed0cc46649d4805ca3bc2bf5fd38fdadd5 not found: ID does not exist" containerID="a915d714ab2d957c45ec795bc46a53ed0cc46649d4805ca3bc2bf5fd38fdadd5"
Sep 19 11:57:24.860952 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.860860    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a915d714ab2d957c45ec795bc46a53ed0cc46649d4805ca3bc2bf5fd38fdadd5"} err="failed to get container status \"a915d714ab2d957c45ec795bc46a53ed0cc46649d4805ca3bc2bf5fd38fdadd5\": rpc error: code = NotFound desc = could not find container \"a915d714ab2d957c45ec795bc46a53ed0cc46649d4805ca3bc2bf5fd38fdadd5\": container with ID starting with a915d714ab2d957c45ec795bc46a53ed0cc46649d4805ca3bc2bf5fd38fdadd5 not found: ID does not exist"
Sep 19 11:57:24.860952 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.860879    2584 scope.go:117] "RemoveContainer" containerID="7db13862736d672e47f320be121ddd06ffa4d5cca24611981e08df799d827ace"
Sep 19 11:57:24.861176 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:57:24.861159    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db13862736d672e47f320be121ddd06ffa4d5cca24611981e08df799d827ace\": container with ID starting with 7db13862736d672e47f320be121ddd06ffa4d5cca24611981e08df799d827ace not found: ID does not exist" containerID="7db13862736d672e47f320be121ddd06ffa4d5cca24611981e08df799d827ace"
Sep 19 11:57:24.861220 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.861183    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db13862736d672e47f320be121ddd06ffa4d5cca24611981e08df799d827ace"} err="failed to get container status \"7db13862736d672e47f320be121ddd06ffa4d5cca24611981e08df799d827ace\": rpc error: code = NotFound desc = could not find container \"7db13862736d672e47f320be121ddd06ffa4d5cca24611981e08df799d827ace\": container with ID starting with 7db13862736d672e47f320be121ddd06ffa4d5cca24611981e08df799d827ace not found: ID does not exist"
Sep 19 11:57:24.861220 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.861199    2584 scope.go:117] "RemoveContainer" containerID="26bc0f6c3d638227dd5217d01bea399a763d12d93cbb978441367afa29d484e7"
Sep 19 11:57:24.866470 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.866441    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"]
Sep 19 11:57:24.870672 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.870644    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7eb15e-8648965985-j2hvl"]
Sep 19 11:57:24.871506 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.871486    2584 scope.go:117] "RemoveContainer" containerID="26bc0f6c3d638227dd5217d01bea399a763d12d93cbb978441367afa29d484e7"
Sep 19 11:57:24.871855 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:57:24.871835    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26bc0f6c3d638227dd5217d01bea399a763d12d93cbb978441367afa29d484e7\": container with ID starting with 26bc0f6c3d638227dd5217d01bea399a763d12d93cbb978441367afa29d484e7 not found: ID does not exist" containerID="26bc0f6c3d638227dd5217d01bea399a763d12d93cbb978441367afa29d484e7"
Sep 19 11:57:24.871924 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.871865    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26bc0f6c3d638227dd5217d01bea399a763d12d93cbb978441367afa29d484e7"} err="failed to get container status \"26bc0f6c3d638227dd5217d01bea399a763d12d93cbb978441367afa29d484e7\": rpc error: code = NotFound desc = could not find container \"26bc0f6c3d638227dd5217d01bea399a763d12d93cbb978441367afa29d484e7\": container with ID starting with 26bc0f6c3d638227dd5217d01bea399a763d12d93cbb978441367afa29d484e7 not found: ID does not exist"
Sep 19 11:57:24.881337 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.881299    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"]
Sep 19 11:57:24.884861 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:24.884835    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-176c86-openshift-default-5c986cf86-5jjcn"]
Sep 19 11:57:25.187527 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:25.187416    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:25.856063 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:25.856026    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:25.904213 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:25.904179    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64ffcacc-383a-4bf4-bb7e-1e03ebafdf65" path="/var/lib/kubelet/pods/64ffcacc-383a-4bf4-bb7e-1e03ebafdf65/volumes"
Sep 19 11:57:25.904636 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:25.904622    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81bbf18e-1913-4789-a8b0-e8bf23d0b99a" path="/var/lib/kubelet/pods/81bbf18e-1913-4789-a8b0-e8bf23d0b99a/volumes"
Sep 19 11:57:26.642691 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:26.642658    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:27.913036 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:27.912990    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g" podUID="7173d3df-0aec-490f-9afa-ed430d9d4a99" containerName="main" containerID="cri-o://edde79a142c65f6a662bb2e5e402823759f2420b405c68e80044da6381b09b26" gracePeriod=45
Sep 19 11:57:27.934442 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:27.934395    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l" podUID="f0abba87-3d88-407c-abce-ba912965c5c5" containerName="main" containerID="cri-o://38db6cbc688f751f927f80f16a4b160cd68dc8ca06ee0eb8cf077065cd78ad1b" gracePeriod=45
Sep 19 11:57:27.944333 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:27.944299    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6" podUID="aad33752-6902-44e6-8d95-7954245f6787" containerName="main" containerID="cri-o://ef5de2b473feb28da2e4e30f04b5350c77bbfa3fe0e76120304c7dc36b009fab" gracePeriod=45
Sep 19 11:57:27.961029 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:27.960995    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g" podUID="7173d3df-0aec-490f-9afa-ed430d9d4a99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 19 11:57:28.267359 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.267329    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:57:28.269521 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.269487    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:57:28.271606 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.271582    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:57:28.342553 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.342521    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfhgf\" (UniqueName: \"kubernetes.io/projected/7173d3df-0aec-490f-9afa-ed430d9d4a99-kube-api-access-tfhgf\") pod \"7173d3df-0aec-490f-9afa-ed430d9d4a99\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") "
Sep 19 11:57:28.342553 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.342565    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-model-cache\") pod \"7173d3df-0aec-490f-9afa-ed430d9d4a99\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") "
Sep 19 11:57:28.342815 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.342623    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-tmp-dir\") pod \"aad33752-6902-44e6-8d95-7954245f6787\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") "
Sep 19 11:57:28.342815 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.342681    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-model-cache\") pod \"f0abba87-3d88-407c-abce-ba912965c5c5\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") "
Sep 19 11:57:28.342815 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.342707    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-tmp-dir\") pod \"7173d3df-0aec-490f-9afa-ed430d9d4a99\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") "
Sep 19 11:57:28.342815 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.342744    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbnrv\" (UniqueName: \"kubernetes.io/projected/f0abba87-3d88-407c-abce-ba912965c5c5-kube-api-access-jbnrv\") pod \"f0abba87-3d88-407c-abce-ba912965c5c5\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") "
Sep 19 11:57:28.342815 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.342773    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-model-cache\") pod \"aad33752-6902-44e6-8d95-7954245f6787\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") "
Sep 19 11:57:28.342815 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.342814    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-dshm\") pod \"7173d3df-0aec-490f-9afa-ed430d9d4a99\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") "
Sep 19 11:57:28.343175 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.342859    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-home\") pod \"f0abba87-3d88-407c-abce-ba912965c5c5\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") "
Sep 19 11:57:28.343175 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.342886    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7173d3df-0aec-490f-9afa-ed430d9d4a99-tls-certs\") pod \"7173d3df-0aec-490f-9afa-ed430d9d4a99\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") "
Sep 19 11:57:28.343175 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.342964    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-db88j\" (UniqueName: \"kubernetes.io/projected/aad33752-6902-44e6-8d95-7954245f6787-kube-api-access-db88j\") pod \"aad33752-6902-44e6-8d95-7954245f6787\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") "
Sep 19 11:57:28.343175 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.342996    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-dshm\") pod \"aad33752-6902-44e6-8d95-7954245f6787\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") "
Sep 19 11:57:28.343175 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.343045    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-home\") pod \"7173d3df-0aec-490f-9afa-ed430d9d4a99\" (UID: \"7173d3df-0aec-490f-9afa-ed430d9d4a99\") "
Sep 19 11:57:28.343175 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.343065    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-home\") pod \"aad33752-6902-44e6-8d95-7954245f6787\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") "
Sep 19 11:57:28.343175 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.343092    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-tmp-dir\") pod \"f0abba87-3d88-407c-abce-ba912965c5c5\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") "
Sep 19 11:57:28.343175 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.343061    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7173d3df-0aec-490f-9afa-ed430d9d4a99" (UID: "7173d3df-0aec-490f-9afa-ed430d9d4a99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:57:28.343175 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.343124    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-dshm\") pod \"f0abba87-3d88-407c-abce-ba912965c5c5\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") "
Sep 19 11:57:28.343175 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.343161    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad33752-6902-44e6-8d95-7954245f6787-tls-certs\") pod \"aad33752-6902-44e6-8d95-7954245f6787\" (UID: \"aad33752-6902-44e6-8d95-7954245f6787\") "
Sep 19 11:57:28.343701 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.343176    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0abba87-3d88-407c-abce-ba912965c5c5" (UID: "f0abba87-3d88-407c-abce-ba912965c5c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:57:28.343701 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.343198    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0abba87-3d88-407c-abce-ba912965c5c5-tls-certs\") pod \"f0abba87-3d88-407c-abce-ba912965c5c5\" (UID: \"f0abba87-3d88-407c-abce-ba912965c5c5\") "
Sep 19 11:57:28.343701 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.343212    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-model-cache" (OuterVolumeSpecName: "model-cache") pod "7173d3df-0aec-490f-9afa-ed430d9d4a99" (UID: "7173d3df-0aec-490f-9afa-ed430d9d4a99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:57:28.343701 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.343500    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-model-cache" (OuterVolumeSpecName: "model-cache") pod "aad33752-6902-44e6-8d95-7954245f6787" (UID: "aad33752-6902-44e6-8d95-7954245f6787"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:57:28.343701 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.343517    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-model-cache\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:28.343701 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.343536    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-model-cache\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:28.343701 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.343549    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-tmp-dir\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:28.343701 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.343656    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-home" (OuterVolumeSpecName: "home") pod "aad33752-6902-44e6-8d95-7954245f6787" (UID: "aad33752-6902-44e6-8d95-7954245f6787"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:57:28.343701 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.343678    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aad33752-6902-44e6-8d95-7954245f6787" (UID: "aad33752-6902-44e6-8d95-7954245f6787"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:57:28.344337 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.344063    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-home" (OuterVolumeSpecName: "home") pod "7173d3df-0aec-490f-9afa-ed430d9d4a99" (UID: "7173d3df-0aec-490f-9afa-ed430d9d4a99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:57:28.347235 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.347032    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-dshm" (OuterVolumeSpecName: "dshm") pod "aad33752-6902-44e6-8d95-7954245f6787" (UID: "aad33752-6902-44e6-8d95-7954245f6787"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:57:28.347235 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.347037    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0abba87-3d88-407c-abce-ba912965c5c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0abba87-3d88-407c-abce-ba912965c5c5" (UID: "f0abba87-3d88-407c-abce-ba912965c5c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:57:28.347445 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.347318    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad33752-6902-44e6-8d95-7954245f6787-kube-api-access-db88j" (OuterVolumeSpecName: "kube-api-access-db88j") pod "aad33752-6902-44e6-8d95-7954245f6787" (UID: "aad33752-6902-44e6-8d95-7954245f6787"). InnerVolumeSpecName "kube-api-access-db88j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:57:28.347586 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.347508    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0abba87-3d88-407c-abce-ba912965c5c5" (UID: "f0abba87-3d88-407c-abce-ba912965c5c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:57:28.347586 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.347553    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-home" (OuterVolumeSpecName: "home") pod "f0abba87-3d88-407c-abce-ba912965c5c5" (UID: "f0abba87-3d88-407c-abce-ba912965c5c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:57:28.347728 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.347613    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7173d3df-0aec-490f-9afa-ed430d9d4a99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7173d3df-0aec-490f-9afa-ed430d9d4a99" (UID: "7173d3df-0aec-490f-9afa-ed430d9d4a99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:57:28.347728 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.347613    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0abba87-3d88-407c-abce-ba912965c5c5-kube-api-access-jbnrv" (OuterVolumeSpecName: "kube-api-access-jbnrv") pod "f0abba87-3d88-407c-abce-ba912965c5c5" (UID: "f0abba87-3d88-407c-abce-ba912965c5c5"). InnerVolumeSpecName "kube-api-access-jbnrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:57:28.348251 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.348187    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-dshm" (OuterVolumeSpecName: "dshm") pod "f0abba87-3d88-407c-abce-ba912965c5c5" (UID: "f0abba87-3d88-407c-abce-ba912965c5c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:57:28.348251 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.348219    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7173d3df-0aec-490f-9afa-ed430d9d4a99-kube-api-access-tfhgf" (OuterVolumeSpecName: "kube-api-access-tfhgf") pod "7173d3df-0aec-490f-9afa-ed430d9d4a99" (UID: "7173d3df-0aec-490f-9afa-ed430d9d4a99"). InnerVolumeSpecName "kube-api-access-tfhgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:57:28.349136 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.349112    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-dshm" (OuterVolumeSpecName: "dshm") pod "7173d3df-0aec-490f-9afa-ed430d9d4a99" (UID: "7173d3df-0aec-490f-9afa-ed430d9d4a99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:57:28.349464 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.349446    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aad33752-6902-44e6-8d95-7954245f6787-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aad33752-6902-44e6-8d95-7954245f6787" (UID: "aad33752-6902-44e6-8d95-7954245f6787"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:57:28.444266 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.444224    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbnrv\" (UniqueName: \"kubernetes.io/projected/f0abba87-3d88-407c-abce-ba912965c5c5-kube-api-access-jbnrv\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:28.444266 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.444261    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-model-cache\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:28.444266 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.444277    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-dshm\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:28.444509 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.444287    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-home\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:28.444509 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.444295    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7173d3df-0aec-490f-9afa-ed430d9d4a99-tls-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:28.444509 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.444302    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db88j\" (UniqueName: \"kubernetes.io/projected/aad33752-6902-44e6-8d95-7954245f6787-kube-api-access-db88j\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:28.444509 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.444311    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-dshm\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:28.444509 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.444321    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7173d3df-0aec-490f-9afa-ed430d9d4a99-home\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:28.444509 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.444327    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-home\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:28.444509 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.444337    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-tmp-dir\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:28.444509 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.444347    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0abba87-3d88-407c-abce-ba912965c5c5-dshm\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:28.444509 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.444359    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad33752-6902-44e6-8d95-7954245f6787-tls-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:28.444509 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.444368    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0abba87-3d88-407c-abce-ba912965c5c5-tls-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:28.444509 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.444376    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfhgf\" (UniqueName: \"kubernetes.io/projected/7173d3df-0aec-490f-9afa-ed430d9d4a99-kube-api-access-tfhgf\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:28.444509 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.444382    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad33752-6902-44e6-8d95-7954245f6787-tmp-dir\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:57:28.866202 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.866162    2584 generic.go:356] "Generic (PLEG): container finished" podID="f0abba87-3d88-407c-abce-ba912965c5c5" containerID="38db6cbc688f751f927f80f16a4b160cd68dc8ca06ee0eb8cf077065cd78ad1b" exitCode=0
Sep 19 11:57:28.866394 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.866213    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"
Sep 19 11:57:28.866394 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.866248    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l" event={"ID":"f0abba87-3d88-407c-abce-ba912965c5c5","Type":"ContainerDied","Data":"38db6cbc688f751f927f80f16a4b160cd68dc8ca06ee0eb8cf077065cd78ad1b"}
Sep 19 11:57:28.866394 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.866296    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l" event={"ID":"f0abba87-3d88-407c-abce-ba912965c5c5","Type":"ContainerDied","Data":"bd1c504ebec125a3735a50ed118a5bb046124302aa79e03b34c2824e49107bea"}
Sep 19 11:57:28.866394 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.866321    2584 scope.go:117] "RemoveContainer" containerID="38db6cbc688f751f927f80f16a4b160cd68dc8ca06ee0eb8cf077065cd78ad1b"
Sep 19 11:57:28.867605 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.867576    2584 generic.go:356] "Generic (PLEG): container finished" podID="aad33752-6902-44e6-8d95-7954245f6787" containerID="ef5de2b473feb28da2e4e30f04b5350c77bbfa3fe0e76120304c7dc36b009fab" exitCode=0
Sep 19 11:57:28.867747 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.867613    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6" event={"ID":"aad33752-6902-44e6-8d95-7954245f6787","Type":"ContainerDied","Data":"ef5de2b473feb28da2e4e30f04b5350c77bbfa3fe0e76120304c7dc36b009fab"}
Sep 19 11:57:28.867747 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.867636    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"
Sep 19 11:57:28.867747 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.867659    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6" event={"ID":"aad33752-6902-44e6-8d95-7954245f6787","Type":"ContainerDied","Data":"b6c854dc0f1fab2fe01891dde491d2d4fb717fd5085d9be44649e3c1fbcec429"}
Sep 19 11:57:28.869225 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.869171    2584 generic.go:356] "Generic (PLEG): container finished" podID="7173d3df-0aec-490f-9afa-ed430d9d4a99" containerID="edde79a142c65f6a662bb2e5e402823759f2420b405c68e80044da6381b09b26" exitCode=0
Sep 19 11:57:28.869335 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.869296    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"
Sep 19 11:57:28.869335 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.869322    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g" event={"ID":"7173d3df-0aec-490f-9afa-ed430d9d4a99","Type":"ContainerDied","Data":"edde79a142c65f6a662bb2e5e402823759f2420b405c68e80044da6381b09b26"}
Sep 19 11:57:28.869454 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.869355    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g" event={"ID":"7173d3df-0aec-490f-9afa-ed430d9d4a99","Type":"ContainerDied","Data":"b24b9bea91bbcc199db0773b3091080721dcdbaef4abe2bbbe30bc6f9e615714"}
Sep 19 11:57:28.877901 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.877879    2584 scope.go:117] "RemoveContainer" containerID="38db6cbc688f751f927f80f16a4b160cd68dc8ca06ee0eb8cf077065cd78ad1b"
Sep 19 11:57:28.878265 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:57:28.878222    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38db6cbc688f751f927f80f16a4b160cd68dc8ca06ee0eb8cf077065cd78ad1b\": container with ID starting with 38db6cbc688f751f927f80f16a4b160cd68dc8ca06ee0eb8cf077065cd78ad1b not found: ID does not exist" containerID="38db6cbc688f751f927f80f16a4b160cd68dc8ca06ee0eb8cf077065cd78ad1b"
Sep 19 11:57:28.878380 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.878257    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38db6cbc688f751f927f80f16a4b160cd68dc8ca06ee0eb8cf077065cd78ad1b"} err="failed to get container status \"38db6cbc688f751f927f80f16a4b160cd68dc8ca06ee0eb8cf077065cd78ad1b\": rpc error: code = NotFound desc = could not find container \"38db6cbc688f751f927f80f16a4b160cd68dc8ca06ee0eb8cf077065cd78ad1b\": container with ID starting with 38db6cbc688f751f927f80f16a4b160cd68dc8ca06ee0eb8cf077065cd78ad1b not found: ID does not exist"
Sep 19 11:57:28.878380 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.878280    2584 scope.go:117] "RemoveContainer" containerID="ef5de2b473feb28da2e4e30f04b5350c77bbfa3fe0e76120304c7dc36b009fab"
Sep 19 11:57:28.887482 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.887464    2584 scope.go:117] "RemoveContainer" containerID="ef5de2b473feb28da2e4e30f04b5350c77bbfa3fe0e76120304c7dc36b009fab"
Sep 19 11:57:28.887789 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:57:28.887769    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef5de2b473feb28da2e4e30f04b5350c77bbfa3fe0e76120304c7dc36b009fab\": container with ID starting with ef5de2b473feb28da2e4e30f04b5350c77bbfa3fe0e76120304c7dc36b009fab not found: ID does not exist" containerID="ef5de2b473feb28da2e4e30f04b5350c77bbfa3fe0e76120304c7dc36b009fab"
Sep 19 11:57:28.887840 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.887795    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef5de2b473feb28da2e4e30f04b5350c77bbfa3fe0e76120304c7dc36b009fab"} err="failed to get container status \"ef5de2b473feb28da2e4e30f04b5350c77bbfa3fe0e76120304c7dc36b009fab\": rpc error: code = NotFound desc = could not find container \"ef5de2b473feb28da2e4e30f04b5350c77bbfa3fe0e76120304c7dc36b009fab\": container with ID starting with ef5de2b473feb28da2e4e30f04b5350c77bbfa3fe0e76120304c7dc36b009fab not found: ID does not exist"
Sep 19 11:57:28.887840 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.887816    2584 scope.go:117] "RemoveContainer" containerID="edde79a142c65f6a662bb2e5e402823759f2420b405c68e80044da6381b09b26"
Sep 19 11:57:28.888872 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.888854    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"]
Sep 19 11:57:28.892302 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.892281    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7eb15e/premium-w0-7eb15e-kserve-77b4884578-76g4l"]
Sep 19 11:57:28.896549 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.896533    2584 scope.go:117] "RemoveContainer" containerID="edde79a142c65f6a662bb2e5e402823759f2420b405c68e80044da6381b09b26"
Sep 19 11:57:28.896844 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:57:28.896828    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edde79a142c65f6a662bb2e5e402823759f2420b405c68e80044da6381b09b26\": container with ID starting with edde79a142c65f6a662bb2e5e402823759f2420b405c68e80044da6381b09b26 not found: ID does not exist" containerID="edde79a142c65f6a662bb2e5e402823759f2420b405c68e80044da6381b09b26"
Sep 19 11:57:28.896891 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.896852    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edde79a142c65f6a662bb2e5e402823759f2420b405c68e80044da6381b09b26"} err="failed to get container status \"edde79a142c65f6a662bb2e5e402823759f2420b405c68e80044da6381b09b26\": rpc error: code = NotFound desc = could not find container \"edde79a142c65f6a662bb2e5e402823759f2420b405c68e80044da6381b09b26\": container with ID starting with edde79a142c65f6a662bb2e5e402823759f2420b405c68e80044da6381b09b26 not found: ID does not exist"
Sep 19 11:57:28.902030 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.902009    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"]
Sep 19 11:57:28.907137 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.907116    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7eb15e/simulator-w0-7eb15e-kserve-6988dcb997-7vkc6"]
Sep 19 11:57:28.918757 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.918732    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"]
Sep 19 11:57:28.922005 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:28.921983    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7eb15e/distinct-2-w0-7eb15e-kserve-757594777b-kvv2g"]
Sep 19 11:57:29.904377 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:29.904347    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7173d3df-0aec-490f-9afa-ed430d9d4a99" path="/var/lib/kubelet/pods/7173d3df-0aec-490f-9afa-ed430d9d4a99/volumes"
Sep 19 11:57:29.904758 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:29.904744    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aad33752-6902-44e6-8d95-7954245f6787" path="/var/lib/kubelet/pods/aad33752-6902-44e6-8d95-7954245f6787/volumes"
Sep 19 11:57:29.905125 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:29.905114    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0abba87-3d88-407c-abce-ba912965c5c5" path="/var/lib/kubelet/pods/f0abba87-3d88-407c-abce-ba912965c5c5/volumes"
Sep 19 11:57:30.040931 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:30.040881    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:30.094474 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:30.094436    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:30.299121 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:30.299022    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:49.067537 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:49.067506    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:49.167750 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:49.167719    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:50.342524 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:50.342489    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:54.336411 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:54.336374    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"]
Sep 19 11:57:54.362169 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:54.362138    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:55.751120 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:55.751091    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:57:58.745394 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:58.745356    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"]
Sep 19 11:57:58.760702 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:57:58.760672    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:58:09.359122 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.359048    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn" podUID="f8bf92f3-1293-4d69-b8f9-4aa49aa924be" containerName="main" containerID="cri-o://b33d3cb8f352b155017541160f2d5f734c0d1ea9fd2df0b0b68c0f9362611181" gracePeriod=45
Sep 19 11:58:09.621390 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.621315    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:58:09.745592 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.745552    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-model-cache\") pod \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") "
Sep 19 11:58:09.745807 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.745633    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-tmp-dir\") pod \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") "
Sep 19 11:58:09.745807 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.745698    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-home\") pod \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") "
Sep 19 11:58:09.745807 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.745740    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-dshm\") pod \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") "
Sep 19 11:58:09.745807 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.745783    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmfg9\" (UniqueName: \"kubernetes.io/projected/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-kube-api-access-zmfg9\") pod \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") "
Sep 19 11:58:09.746080 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.745822    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-tls-certs\") pod \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\" (UID: \"f8bf92f3-1293-4d69-b8f9-4aa49aa924be\") "
Sep 19 11:58:09.746080 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.745860    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-model-cache" (OuterVolumeSpecName: "model-cache") pod "f8bf92f3-1293-4d69-b8f9-4aa49aa924be" (UID: "f8bf92f3-1293-4d69-b8f9-4aa49aa924be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:58:09.746080 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.745959    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-home" (OuterVolumeSpecName: "home") pod "f8bf92f3-1293-4d69-b8f9-4aa49aa924be" (UID: "f8bf92f3-1293-4d69-b8f9-4aa49aa924be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:58:09.746080 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.745967    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f8bf92f3-1293-4d69-b8f9-4aa49aa924be" (UID: "f8bf92f3-1293-4d69-b8f9-4aa49aa924be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:58:09.746299 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.746200    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-model-cache\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:58:09.746299 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.746225    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-tmp-dir\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:58:09.746299 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.746241    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-home\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:58:09.748250 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.748220    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-dshm" (OuterVolumeSpecName: "dshm") pod "f8bf92f3-1293-4d69-b8f9-4aa49aa924be" (UID: "f8bf92f3-1293-4d69-b8f9-4aa49aa924be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:58:09.748250 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.748215    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8bf92f3-1293-4d69-b8f9-4aa49aa924be" (UID: "f8bf92f3-1293-4d69-b8f9-4aa49aa924be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:58:09.748402 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.748278    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-kube-api-access-zmfg9" (OuterVolumeSpecName: "kube-api-access-zmfg9") pod "f8bf92f3-1293-4d69-b8f9-4aa49aa924be" (UID: "f8bf92f3-1293-4d69-b8f9-4aa49aa924be"). InnerVolumeSpecName "kube-api-access-zmfg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:58:09.847287 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.847237    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-tls-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:58:09.847287 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.847279    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-dshm\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:58:09.847287 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:09.847292    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmfg9\" (UniqueName: \"kubernetes.io/projected/f8bf92f3-1293-4d69-b8f9-4aa49aa924be-kube-api-access-zmfg9\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:58:10.096972 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:10.096900    2584 generic.go:356] "Generic (PLEG): container finished" podID="f8bf92f3-1293-4d69-b8f9-4aa49aa924be" containerID="b33d3cb8f352b155017541160f2d5f734c0d1ea9fd2df0b0b68c0f9362611181" exitCode=0
Sep 19 11:58:10.096972 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:10.096961    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"
Sep 19 11:58:10.097248 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:10.096989    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn" event={"ID":"f8bf92f3-1293-4d69-b8f9-4aa49aa924be","Type":"ContainerDied","Data":"b33d3cb8f352b155017541160f2d5f734c0d1ea9fd2df0b0b68c0f9362611181"}
Sep 19 11:58:10.097248 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:10.097035    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn" event={"ID":"f8bf92f3-1293-4d69-b8f9-4aa49aa924be","Type":"ContainerDied","Data":"5d788928a897956c532320cd167b0ab2d7abf7539577a37aefff64e983185f9c"}
Sep 19 11:58:10.097248 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:10.097057    2584 scope.go:117] "RemoveContainer" containerID="b33d3cb8f352b155017541160f2d5f734c0d1ea9fd2df0b0b68c0f9362611181"
Sep 19 11:58:10.107065 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:10.107038    2584 scope.go:117] "RemoveContainer" containerID="b33d3cb8f352b155017541160f2d5f734c0d1ea9fd2df0b0b68c0f9362611181"
Sep 19 11:58:10.107422 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:58:10.107402    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33d3cb8f352b155017541160f2d5f734c0d1ea9fd2df0b0b68c0f9362611181\": container with ID starting with b33d3cb8f352b155017541160f2d5f734c0d1ea9fd2df0b0b68c0f9362611181 not found: ID does not exist" containerID="b33d3cb8f352b155017541160f2d5f734c0d1ea9fd2df0b0b68c0f9362611181"
Sep 19 11:58:10.107471 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:10.107432    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33d3cb8f352b155017541160f2d5f734c0d1ea9fd2df0b0b68c0f9362611181"} err="failed to get container status \"b33d3cb8f352b155017541160f2d5f734c0d1ea9fd2df0b0b68c0f9362611181\": rpc error: code = NotFound desc = could not find container \"b33d3cb8f352b155017541160f2d5f734c0d1ea9fd2df0b0b68c0f9362611181\": container with ID starting with b33d3cb8f352b155017541160f2d5f734c0d1ea9fd2df0b0b68c0f9362611181 not found: ID does not exist"
Sep 19 11:58:10.113233 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:10.113200    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"]
Sep 19 11:58:10.115236 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:10.115208    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4937ff/auth-test-model-4937ff-kserve-59889cbcf6-lvfjn"]
Sep 19 11:58:11.904619 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:11.904584    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8bf92f3-1293-4d69-b8f9-4aa49aa924be" path="/var/lib/kubelet/pods/f8bf92f3-1293-4d69-b8f9-4aa49aa924be/volumes"
Sep 19 11:58:13.766173 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:13.766105    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4" podUID="e11958f3-990d-45af-9a32-ae08291ec3e2" containerName="main" containerID="cri-o://9b97c1a215cd6e56dff44b52ed88c336ee679bd891245f4025f7e3469e95583e" gracePeriod=45
Sep 19 11:58:14.038356 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.038330    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:58:14.086578 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.086533    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-home\") pod \"e11958f3-990d-45af-9a32-ae08291ec3e2\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") "
Sep 19 11:58:14.086739 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.086608    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-dshm\") pod \"e11958f3-990d-45af-9a32-ae08291ec3e2\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") "
Sep 19 11:58:14.086739 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.086638    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-model-cache\") pod \"e11958f3-990d-45af-9a32-ae08291ec3e2\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") "
Sep 19 11:58:14.086739 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.086657    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn4q8\" (UniqueName: \"kubernetes.io/projected/e11958f3-990d-45af-9a32-ae08291ec3e2-kube-api-access-tn4q8\") pod \"e11958f3-990d-45af-9a32-ae08291ec3e2\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") "
Sep 19 11:58:14.086739 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.086680    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-tmp-dir\") pod \"e11958f3-990d-45af-9a32-ae08291ec3e2\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") "
Sep 19 11:58:14.086739 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.086730    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e11958f3-990d-45af-9a32-ae08291ec3e2-tls-certs\") pod \"e11958f3-990d-45af-9a32-ae08291ec3e2\" (UID: \"e11958f3-990d-45af-9a32-ae08291ec3e2\") "
Sep 19 11:58:14.087066 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.086844    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-home" (OuterVolumeSpecName: "home") pod "e11958f3-990d-45af-9a32-ae08291ec3e2" (UID: "e11958f3-990d-45af-9a32-ae08291ec3e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:58:14.087066 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.086947    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "e11958f3-990d-45af-9a32-ae08291ec3e2" (UID: "e11958f3-990d-45af-9a32-ae08291ec3e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:58:14.087177 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.087079    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-home\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:58:14.087177 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.087099    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-model-cache\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:58:14.087177 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.087074    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e11958f3-990d-45af-9a32-ae08291ec3e2" (UID: "e11958f3-990d-45af-9a32-ae08291ec3e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:58:14.089181 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.089152    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-dshm" (OuterVolumeSpecName: "dshm") pod "e11958f3-990d-45af-9a32-ae08291ec3e2" (UID: "e11958f3-990d-45af-9a32-ae08291ec3e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:58:14.089322 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.089228    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e11958f3-990d-45af-9a32-ae08291ec3e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e11958f3-990d-45af-9a32-ae08291ec3e2" (UID: "e11958f3-990d-45af-9a32-ae08291ec3e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:58:14.089373 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.089319    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11958f3-990d-45af-9a32-ae08291ec3e2-kube-api-access-tn4q8" (OuterVolumeSpecName: "kube-api-access-tn4q8") pod "e11958f3-990d-45af-9a32-ae08291ec3e2" (UID: "e11958f3-990d-45af-9a32-ae08291ec3e2"). InnerVolumeSpecName "kube-api-access-tn4q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:58:14.119264 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.119226    2584 generic.go:356] "Generic (PLEG): container finished" podID="e11958f3-990d-45af-9a32-ae08291ec3e2" containerID="9b97c1a215cd6e56dff44b52ed88c336ee679bd891245f4025f7e3469e95583e" exitCode=0
Sep 19 11:58:14.119264 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.119275    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"
Sep 19 11:58:14.119507 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.119306    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4" event={"ID":"e11958f3-990d-45af-9a32-ae08291ec3e2","Type":"ContainerDied","Data":"9b97c1a215cd6e56dff44b52ed88c336ee679bd891245f4025f7e3469e95583e"}
Sep 19 11:58:14.119507 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.119357    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4" event={"ID":"e11958f3-990d-45af-9a32-ae08291ec3e2","Type":"ContainerDied","Data":"dbb763a024ff1ed6b8348e6e66c60b2a9002ecc8d00f9dee6009af548526b4d4"}
Sep 19 11:58:14.119507 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.119375    2584 scope.go:117] "RemoveContainer" containerID="9b97c1a215cd6e56dff44b52ed88c336ee679bd891245f4025f7e3469e95583e"
Sep 19 11:58:14.128847 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.128828    2584 scope.go:117] "RemoveContainer" containerID="9b97c1a215cd6e56dff44b52ed88c336ee679bd891245f4025f7e3469e95583e"
Sep 19 11:58:14.129224 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:58:14.129201    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b97c1a215cd6e56dff44b52ed88c336ee679bd891245f4025f7e3469e95583e\": container with ID starting with 9b97c1a215cd6e56dff44b52ed88c336ee679bd891245f4025f7e3469e95583e not found: ID does not exist" containerID="9b97c1a215cd6e56dff44b52ed88c336ee679bd891245f4025f7e3469e95583e"
Sep 19 11:58:14.129326 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.129233    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b97c1a215cd6e56dff44b52ed88c336ee679bd891245f4025f7e3469e95583e"} err="failed to get container status \"9b97c1a215cd6e56dff44b52ed88c336ee679bd891245f4025f7e3469e95583e\": rpc error: code = NotFound desc = could not find container \"9b97c1a215cd6e56dff44b52ed88c336ee679bd891245f4025f7e3469e95583e\": container with ID starting with 9b97c1a215cd6e56dff44b52ed88c336ee679bd891245f4025f7e3469e95583e not found: ID does not exist"
Sep 19 11:58:14.139892 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.139862    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"]
Sep 19 11:58:14.143219 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.143179    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a8a59/auth-test-model-4a8a59-kserve-74dffd8d8-qlql4"]
Sep 19 11:58:14.188192 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.188161    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn4q8\" (UniqueName: \"kubernetes.io/projected/e11958f3-990d-45af-9a32-ae08291ec3e2-kube-api-access-tn4q8\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:58:14.188192 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.188191    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-tmp-dir\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:58:14.188371 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.188206    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e11958f3-990d-45af-9a32-ae08291ec3e2-tls-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:58:14.188371 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:14.188217    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e11958f3-990d-45af-9a32-ae08291ec3e2-dshm\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:58:15.903481 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:15.903440    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e11958f3-990d-45af-9a32-ae08291ec3e2" path="/var/lib/kubelet/pods/e11958f3-990d-45af-9a32-ae08291ec3e2/volumes"
Sep 19 11:58:16.822867 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.822826    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"]
Sep 19 11:58:16.823211 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823197    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ffcacc-383a-4bf4-bb7e-1e03ebafdf65" containerName="maas-api"
Sep 19 11:58:16.823211 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823211    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ffcacc-383a-4bf4-bb7e-1e03ebafdf65" containerName="maas-api"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823222    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7173d3df-0aec-490f-9afa-ed430d9d4a99" containerName="main"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823228    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7173d3df-0aec-490f-9afa-ed430d9d4a99" containerName="main"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823236    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81bbf18e-1913-4789-a8b0-e8bf23d0b99a" containerName="istio-proxy"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823241    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="81bbf18e-1913-4789-a8b0-e8bf23d0b99a" containerName="istio-proxy"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823248    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d83f095e-42c9-4369-9e63-e29edc8ddd54" containerName="istio-proxy"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823253    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d83f095e-42c9-4369-9e63-e29edc8ddd54" containerName="istio-proxy"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823260    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8bf92f3-1293-4d69-b8f9-4aa49aa924be" containerName="main"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823265    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8bf92f3-1293-4d69-b8f9-4aa49aa924be" containerName="main"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823276    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81bbf18e-1913-4789-a8b0-e8bf23d0b99a" containerName="istio-proxy"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823281    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="81bbf18e-1913-4789-a8b0-e8bf23d0b99a" containerName="istio-proxy"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823289    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e11958f3-990d-45af-9a32-ae08291ec3e2" containerName="main"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823295    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e11958f3-990d-45af-9a32-ae08291ec3e2" containerName="main"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823300    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="839984cb-3e33-4182-b323-751344047d86" containerName="istio-proxy"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823304    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="839984cb-3e33-4182-b323-751344047d86" containerName="istio-proxy"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823312    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad33752-6902-44e6-8d95-7954245f6787" containerName="main"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823317    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad33752-6902-44e6-8d95-7954245f6787" containerName="main"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823325    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2854ecea-0ccb-4ddc-ac6d-1f8a601d2789" containerName="maas-api"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823330    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2854ecea-0ccb-4ddc-ac6d-1f8a601d2789" containerName="maas-api"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823338    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0abba87-3d88-407c-abce-ba912965c5c5" containerName="main"
Sep 19 11:58:16.823341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823342    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0abba87-3d88-407c-abce-ba912965c5c5" containerName="main"
Sep 19 11:58:16.824400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823400    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="81bbf18e-1913-4789-a8b0-e8bf23d0b99a" containerName="istio-proxy"
Sep 19 11:58:16.824400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823409    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="81bbf18e-1913-4789-a8b0-e8bf23d0b99a" containerName="istio-proxy"
Sep 19 11:58:16.824400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823416    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0abba87-3d88-407c-abce-ba912965c5c5" containerName="main"
Sep 19 11:58:16.824400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823422    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2854ecea-0ccb-4ddc-ac6d-1f8a601d2789" containerName="maas-api"
Sep 19 11:58:16.824400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823427    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d83f095e-42c9-4369-9e63-e29edc8ddd54" containerName="istio-proxy"
Sep 19 11:58:16.824400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823432    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="839984cb-3e33-4182-b323-751344047d86" containerName="istio-proxy"
Sep 19 11:58:16.824400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823438    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7173d3df-0aec-490f-9afa-ed430d9d4a99" containerName="main"
Sep 19 11:58:16.824400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823444    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ffcacc-383a-4bf4-bb7e-1e03ebafdf65" containerName="maas-api"
Sep 19 11:58:16.824400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823449    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8bf92f3-1293-4d69-b8f9-4aa49aa924be" containerName="main"
Sep 19 11:58:16.824400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823455    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e11958f3-990d-45af-9a32-ae08291ec3e2" containerName="main"
Sep 19 11:58:16.824400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823459    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="839984cb-3e33-4182-b323-751344047d86" containerName="istio-proxy"
Sep 19 11:58:16.824400 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.823464    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad33752-6902-44e6-8d95-7954245f6787" containerName="main"
Sep 19 11:58:16.827815 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.827790    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:16.831759 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.831734    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4937ff\"/\"default-dockercfg-7njzd\""
Sep 19 11:58:16.831931 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.831803    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4937ff\"/\"sub-test-model-4937ff-kserve-self-signed-certs\""
Sep 19 11:58:16.831931 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.831825    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4937ff\"/\"kube-root-ca.crt\""
Sep 19 11:58:16.831931 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.831743    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4937ff\"/\"openshift-service-ca.crt\""
Sep 19 11:58:16.836150 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.835936    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"]
Sep 19 11:58:16.915883 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.915824    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl7k8\" (UniqueName: \"kubernetes.io/projected/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-kube-api-access-hl7k8\") pod \"sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:16.916326 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.915950    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-model-cache\") pod \"sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:16.916326 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.915989    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-tmp-dir\") pod \"sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:16.916326 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.916012    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-home\") pod \"sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:16.916326 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.916043    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-dshm\") pod \"sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:16.916326 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:16.916120    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-tls-certs\") pod \"sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:17.017448 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:17.017402    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-model-cache\") pod \"sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:17.017448 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:17.017455    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-tmp-dir\") pod \"sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:17.017696 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:17.017487    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-home\") pod \"sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:17.017696 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:17.017517    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-dshm\") pod \"sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:17.017696 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:17.017553    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-tls-certs\") pod \"sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:17.017696 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:17.017655    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl7k8\" (UniqueName: \"kubernetes.io/projected/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-kube-api-access-hl7k8\") pod \"sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:17.017976 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:17.017958    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-model-cache\") pod \"sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:17.018035 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:17.017993    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-tmp-dir\") pod \"sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:17.018109 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:17.018076    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-home\") pod \"sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:17.020159 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:17.020130    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-dshm\") pod \"sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:17.020462 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:17.020445    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-tls-certs\") pod \"sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:17.025327 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:17.025303    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl7k8\" (UniqueName: \"kubernetes.io/projected/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-kube-api-access-hl7k8\") pod \"sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:17.139645 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:17.139544    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:17.416487 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:17.416460    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"]
Sep 19 11:58:17.418832 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:58:17.418799    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cf5f279_6a4f_4470_b8d0_f9733d0d4a63.slice/crio-6752a3f05e868b721bd600b95212ddd35b13faae23624bc0e63ef91cdf4237bb WatchSource:0}: Error finding container 6752a3f05e868b721bd600b95212ddd35b13faae23624bc0e63ef91cdf4237bb: Status 404 returned error can't find the container with id 6752a3f05e868b721bd600b95212ddd35b13faae23624bc0e63ef91cdf4237bb
Sep 19 11:58:17.499749 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:17.499702    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:58:18.143519 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:18.143484    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q" event={"ID":"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63","Type":"ContainerStarted","Data":"42c1b2e4197abf69a013dd315b30d4058d4bdfedb5c8f336d65841e50113e5a6"}
Sep 19 11:58:18.143983 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:18.143531    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q" event={"ID":"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63","Type":"ContainerStarted","Data":"6752a3f05e868b721bd600b95212ddd35b13faae23624bc0e63ef91cdf4237bb"}
Sep 19 11:58:18.161023 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:18.160960    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q" podStartSLOduration=2.160940107 podStartE2EDuration="2.160940107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:58:18.159408312 +0000 UTC m=+1482.359438112" watchObservedRunningTime="2026-09-19 11:58:18.160940107 +0000 UTC m=+1482.360969965"
Sep 19 11:58:19.140351 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:19.140311    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:27.140727 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:27.140682    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:27.153376 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:27.153344    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:27.210278 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:27.210246    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:58:35.424557 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.424498    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"]
Sep 19 11:58:35.425243 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.425219    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d83f095e-42c9-4369-9e63-e29edc8ddd54" containerName="istio-proxy"
Sep 19 11:58:35.425301 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.425247    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d83f095e-42c9-4369-9e63-e29edc8ddd54" containerName="istio-proxy"
Sep 19 11:58:35.425301 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.425277    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="839984cb-3e33-4182-b323-751344047d86" containerName="istio-proxy"
Sep 19 11:58:35.425301 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.425286    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="839984cb-3e33-4182-b323-751344047d86" containerName="istio-proxy"
Sep 19 11:58:35.425792 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.425400    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d83f095e-42c9-4369-9e63-e29edc8ddd54" containerName="istio-proxy"
Sep 19 11:58:35.429165 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.429145    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:35.433506 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.433248    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4a8a59\"/\"openshift-service-ca.crt\""
Sep 19 11:58:35.433506 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.433271    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4a8a59\"/\"default-dockercfg-7wptg\""
Sep 19 11:58:35.433506 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.433372    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4a8a59\"/\"sub-test-model-4a8a59-kserve-self-signed-certs\""
Sep 19 11:58:35.433506 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.433240    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4a8a59\"/\"kube-root-ca.crt\""
Sep 19 11:58:35.434305 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.434280    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"]
Sep 19 11:58:35.489944 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.489873    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-dshm\") pod \"sub-test-model-4a8a59-kserve-c4977b856-8npfd\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:35.489944 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.489938    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-home\") pod \"sub-test-model-4a8a59-kserve-c4977b856-8npfd\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:35.490219 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.489962    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-tls-certs\") pod \"sub-test-model-4a8a59-kserve-c4977b856-8npfd\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:35.490219 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.490009    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-model-cache\") pod \"sub-test-model-4a8a59-kserve-c4977b856-8npfd\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:35.490219 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.490063    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s58gv\" (UniqueName: \"kubernetes.io/projected/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-kube-api-access-s58gv\") pod \"sub-test-model-4a8a59-kserve-c4977b856-8npfd\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:35.490219 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.490102    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-tmp-dir\") pod \"sub-test-model-4a8a59-kserve-c4977b856-8npfd\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:35.542633 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.542593    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:58:35.590856 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.590811    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s58gv\" (UniqueName: \"kubernetes.io/projected/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-kube-api-access-s58gv\") pod \"sub-test-model-4a8a59-kserve-c4977b856-8npfd\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:35.591085 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.590867    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-tmp-dir\") pod \"sub-test-model-4a8a59-kserve-c4977b856-8npfd\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:35.591085 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.590986    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-dshm\") pod \"sub-test-model-4a8a59-kserve-c4977b856-8npfd\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:35.591085 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.591024    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-home\") pod \"sub-test-model-4a8a59-kserve-c4977b856-8npfd\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:35.591085 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.591047    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-tls-certs\") pod \"sub-test-model-4a8a59-kserve-c4977b856-8npfd\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:35.591085 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.591076    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-model-cache\") pod \"sub-test-model-4a8a59-kserve-c4977b856-8npfd\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:35.591343 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.591319    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-tmp-dir\") pod \"sub-test-model-4a8a59-kserve-c4977b856-8npfd\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:35.591389 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.591374    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-home\") pod \"sub-test-model-4a8a59-kserve-c4977b856-8npfd\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:35.591494 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.591474    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-model-cache\") pod \"sub-test-model-4a8a59-kserve-c4977b856-8npfd\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:35.593514 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.593490    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-dshm\") pod \"sub-test-model-4a8a59-kserve-c4977b856-8npfd\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:35.593771 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.593757    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-tls-certs\") pod \"sub-test-model-4a8a59-kserve-c4977b856-8npfd\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:35.598576 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.598552    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s58gv\" (UniqueName: \"kubernetes.io/projected/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-kube-api-access-s58gv\") pod \"sub-test-model-4a8a59-kserve-c4977b856-8npfd\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:35.742265 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:35.742170    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:36.015580 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:36.015429    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"]
Sep 19 11:58:36.017968 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:58:36.017937    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8d259e5_9aa8_4e6d_aa26_1bda21f9d6ff.slice/crio-6e7c47ed786e2546d1d95ff4ae5175a399a29ed5865e5ae807876acc1a77bd19 WatchSource:0}: Error finding container 6e7c47ed786e2546d1d95ff4ae5175a399a29ed5865e5ae807876acc1a77bd19: Status 404 returned error can't find the container with id 6e7c47ed786e2546d1d95ff4ae5175a399a29ed5865e5ae807876acc1a77bd19
Sep 19 11:58:36.242635 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:36.242597    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd" event={"ID":"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff","Type":"ContainerStarted","Data":"7e295d47a18db3a6c861896e10591d06db159bbe8dddef722525055a2474f5b8"}
Sep 19 11:58:36.242635 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:36.242641    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd" event={"ID":"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff","Type":"ContainerStarted","Data":"6e7c47ed786e2546d1d95ff4ae5175a399a29ed5865e5ae807876acc1a77bd19"}
Sep 19 11:58:36.260687 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:36.260630    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd" podStartSLOduration=1.26061249 podStartE2EDuration="1.26061249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:58:36.258051689 +0000 UTC m=+1500.458081493" watchObservedRunningTime="2026-09-19 11:58:36.26061249 +0000 UTC m=+1500.460642294"
Sep 19 11:58:36.742459 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:36.742417    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:37.967507 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:37.967474    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/1.log"
Sep 19 11:58:37.969210 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:37.969190    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/1.log"
Sep 19 11:58:37.973578 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:37.973554    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6th98_2fed27e0-833f-49a8-b11e-8f61df6e6bd5/ovn-acl-logging/0.log"
Sep 19 11:58:37.974865 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:37.974844    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6th98_2fed27e0-833f-49a8-b11e-8f61df6e6bd5/ovn-acl-logging/0.log"
Sep 19 11:58:44.771749 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:44.771716    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:58:45.742828 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:45.742788    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:45.755361 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:45.755330    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:46.306023 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:46.305981    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:58:56.623283 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:58:56.623195    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:59:15.663742 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:15.663694    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6447-03b5f2"]
Sep 19 11:59:15.667694 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:15.667666    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6447-03b5f2"
Sep 19 11:59:15.670664 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:15.670634    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qhqqh\""
Sep 19 11:59:15.672394 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:15.672364    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6447-03b5f2"]
Sep 19 11:59:15.767894 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:15.767854    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt88x\" (UniqueName: \"kubernetes.io/projected/8f8e260b-4048-4a7b-a405-0db54580cf2c-kube-api-access-zt88x\") pod \"test-curl-6447-03b5f2\" (UID: \"8f8e260b-4048-4a7b-a405-0db54580cf2c\") " pod="opendatahub/test-curl-6447-03b5f2"
Sep 19 11:59:15.868866 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:15.868813    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt88x\" (UniqueName: \"kubernetes.io/projected/8f8e260b-4048-4a7b-a405-0db54580cf2c-kube-api-access-zt88x\") pod \"test-curl-6447-03b5f2\" (UID: \"8f8e260b-4048-4a7b-a405-0db54580cf2c\") " pod="opendatahub/test-curl-6447-03b5f2"
Sep 19 11:59:15.876925 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:15.876883    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt88x\" (UniqueName: \"kubernetes.io/projected/8f8e260b-4048-4a7b-a405-0db54580cf2c-kube-api-access-zt88x\") pod \"test-curl-6447-03b5f2\" (UID: \"8f8e260b-4048-4a7b-a405-0db54580cf2c\") " pod="opendatahub/test-curl-6447-03b5f2"
Sep 19 11:59:15.979357 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:15.979261    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6447-03b5f2"
Sep 19 11:59:16.250961 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:16.250907    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6447-03b5f2"]
Sep 19 11:59:16.253450 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:59:16.253419    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f8e260b_4048_4a7b_a405_0db54580cf2c.slice/crio-90541467aec20feb89e0fe65a451dba276ba1f38fe35902c8ece1bc43db2baa3 WatchSource:0}: Error finding container 90541467aec20feb89e0fe65a451dba276ba1f38fe35902c8ece1bc43db2baa3: Status 404 returned error can't find the container with id 90541467aec20feb89e0fe65a451dba276ba1f38fe35902c8ece1bc43db2baa3
Sep 19 11:59:16.453983 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:16.453940    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6447-03b5f2" event={"ID":"8f8e260b-4048-4a7b-a405-0db54580cf2c","Type":"ContainerStarted","Data":"87e1fa104765b48dff286738580ba8ee8ac13b7f267b03ad3ba78bc9d791a3cf"}
Sep 19 11:59:16.453983 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:16.453994    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6447-03b5f2" event={"ID":"8f8e260b-4048-4a7b-a405-0db54580cf2c","Type":"ContainerStarted","Data":"90541467aec20feb89e0fe65a451dba276ba1f38fe35902c8ece1bc43db2baa3"}
Sep 19 11:59:16.468604 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:16.468547    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-6447-03b5f2" podStartSLOduration=1.468527697 podStartE2EDuration="1.468527697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:59:16.466847913 +0000 UTC m=+1540.666877717" watchObservedRunningTime="2026-09-19 11:59:16.468527697 +0000 UTC m=+1540.668557500"
Sep 19 11:59:17.015436 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:17.015337    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6447-03b5f2"]
Sep 19 11:59:17.021341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:17.021305    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6447-03b5f2"]
Sep 19 11:59:17.136976 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:17.136936    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6447-5a4564"]
Sep 19 11:59:17.137318 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:17.137306    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f8e260b-4048-4a7b-a405-0db54580cf2c" containerName="test-curl-6447-03b5f2"
Sep 19 11:59:17.137375 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:17.137320    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f8e260b-4048-4a7b-a405-0db54580cf2c" containerName="test-curl-6447-03b5f2"
Sep 19 11:59:17.137418 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:17.137393    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f8e260b-4048-4a7b-a405-0db54580cf2c" containerName="test-curl-6447-03b5f2"
Sep 19 11:59:17.140523 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:17.140501    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6447-5a4564"
Sep 19 11:59:17.146686 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:17.146658    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6447-5a4564"]
Sep 19 11:59:17.282972 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:17.282847    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrbkl\" (UniqueName: \"kubernetes.io/projected/906ffcd6-d028-4c6a-9b80-0e27511df2c3-kube-api-access-nrbkl\") pod \"test-curl-6447-5a4564\" (UID: \"906ffcd6-d028-4c6a-9b80-0e27511df2c3\") " pod="opendatahub/test-curl-6447-5a4564"
Sep 19 11:59:17.384058 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:17.384016    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrbkl\" (UniqueName: \"kubernetes.io/projected/906ffcd6-d028-4c6a-9b80-0e27511df2c3-kube-api-access-nrbkl\") pod \"test-curl-6447-5a4564\" (UID: \"906ffcd6-d028-4c6a-9b80-0e27511df2c3\") " pod="opendatahub/test-curl-6447-5a4564"
Sep 19 11:59:17.392357 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:17.392314    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrbkl\" (UniqueName: \"kubernetes.io/projected/906ffcd6-d028-4c6a-9b80-0e27511df2c3-kube-api-access-nrbkl\") pod \"test-curl-6447-5a4564\" (UID: \"906ffcd6-d028-4c6a-9b80-0e27511df2c3\") " pod="opendatahub/test-curl-6447-5a4564"
Sep 19 11:59:17.452263 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:17.452224    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6447-5a4564"
Sep 19 11:59:17.459563 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:59:17.459534    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-6447-03b5f2" container="test-curl-6447-03b5f2"
Sep 19 11:59:17.725107 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:17.725078    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6447-5a4564"]
Sep 19 11:59:17.727521 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:59:17.727490    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod906ffcd6_d028_4c6a_9b80_0e27511df2c3.slice/crio-6ab73117e1309b4c5b6346441f262a3539a67edb2640d4596ed92b8fe8113d6e WatchSource:0}: Error finding container 6ab73117e1309b4c5b6346441f262a3539a67edb2640d4596ed92b8fe8113d6e: Status 404 returned error can't find the container with id 6ab73117e1309b4c5b6346441f262a3539a67edb2640d4596ed92b8fe8113d6e
Sep 19 11:59:17.733299 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:17.733270    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:59:17.780187 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:17.780163    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:59:18.466546 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:18.466505    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6447-5a4564" event={"ID":"906ffcd6-d028-4c6a-9b80-0e27511df2c3","Type":"ContainerStarted","Data":"d0a6393974eb13dfdc9def960bdb11f976e05f23b4f2a9a88efc910d682f3a95"}
Sep 19 11:59:18.466546 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:18.466553    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6447-5a4564" event={"ID":"906ffcd6-d028-4c6a-9b80-0e27511df2c3","Type":"ContainerStarted","Data":"6ab73117e1309b4c5b6346441f262a3539a67edb2640d4596ed92b8fe8113d6e"}
Sep 19 11:59:18.467136 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:18.466553    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-6447-03b5f2" podUID="8f8e260b-4048-4a7b-a405-0db54580cf2c" containerName="test-curl-6447-03b5f2" containerID="cri-o://87e1fa104765b48dff286738580ba8ee8ac13b7f267b03ad3ba78bc9d791a3cf" gracePeriod=2
Sep 19 11:59:18.481280 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:18.481223    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-6447-5a4564" podStartSLOduration=1.481204819 podStartE2EDuration="1.481204819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:59:18.479413893 +0000 UTC m=+1542.679443696" watchObservedRunningTime="2026-09-19 11:59:18.481204819 +0000 UTC m=+1542.681234621"
Sep 19 11:59:19.032275 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:19.032237    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6447-5a4564"]
Sep 19 11:59:19.036407 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:19.036364    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6447-5a4564"]
Sep 19 11:59:19.472151 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:59:19.472118    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-6447-5a4564" container="test-curl-6447-5a4564"
Sep 19 11:59:19.874574 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:19.874538    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:59:20.476790 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:20.476753    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-6447-5a4564" podUID="906ffcd6-d028-4c6a-9b80-0e27511df2c3" containerName="test-curl-6447-5a4564" containerID="cri-o://d0a6393974eb13dfdc9def960bdb11f976e05f23b4f2a9a88efc910d682f3a95" gracePeriod=2
Sep 19 11:59:20.644771 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:20.644744    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6447-03b5f2"
Sep 19 11:59:20.816055 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:20.816017    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zt88x\" (UniqueName: \"kubernetes.io/projected/8f8e260b-4048-4a7b-a405-0db54580cf2c-kube-api-access-zt88x\") pod \"8f8e260b-4048-4a7b-a405-0db54580cf2c\" (UID: \"8f8e260b-4048-4a7b-a405-0db54580cf2c\") "
Sep 19 11:59:20.818378 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:20.818341    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8e260b-4048-4a7b-a405-0db54580cf2c-kube-api-access-zt88x" (OuterVolumeSpecName: "kube-api-access-zt88x") pod "8f8e260b-4048-4a7b-a405-0db54580cf2c" (UID: "8f8e260b-4048-4a7b-a405-0db54580cf2c"). InnerVolumeSpecName "kube-api-access-zt88x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:59:20.917020 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:20.916983    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt88x\" (UniqueName: \"kubernetes.io/projected/8f8e260b-4048-4a7b-a405-0db54580cf2c-kube-api-access-zt88x\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:59:21.142038 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:21.141958    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"]
Sep 19 11:59:21.148755 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:21.148730    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:59:21.483007 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:21.482903    2584 generic.go:356] "Generic (PLEG): container finished" podID="8f8e260b-4048-4a7b-a405-0db54580cf2c" containerID="87e1fa104765b48dff286738580ba8ee8ac13b7f267b03ad3ba78bc9d791a3cf" exitCode=137
Sep 19 11:59:21.483007 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:21.482956    2584 scope.go:117] "RemoveContainer" containerID="87e1fa104765b48dff286738580ba8ee8ac13b7f267b03ad3ba78bc9d791a3cf"
Sep 19 11:59:21.483007 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:21.482966    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6447-03b5f2"
Sep 19 11:59:21.492661 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:21.492644    2584 scope.go:117] "RemoveContainer" containerID="87e1fa104765b48dff286738580ba8ee8ac13b7f267b03ad3ba78bc9d791a3cf"
Sep 19 11:59:21.493018 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:59:21.492992    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e1fa104765b48dff286738580ba8ee8ac13b7f267b03ad3ba78bc9d791a3cf\": container with ID starting with 87e1fa104765b48dff286738580ba8ee8ac13b7f267b03ad3ba78bc9d791a3cf not found: ID does not exist" containerID="87e1fa104765b48dff286738580ba8ee8ac13b7f267b03ad3ba78bc9d791a3cf"
Sep 19 11:59:21.493114 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:21.493029    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e1fa104765b48dff286738580ba8ee8ac13b7f267b03ad3ba78bc9d791a3cf"} err="failed to get container status \"87e1fa104765b48dff286738580ba8ee8ac13b7f267b03ad3ba78bc9d791a3cf\": rpc error: code = NotFound desc = could not find container \"87e1fa104765b48dff286738580ba8ee8ac13b7f267b03ad3ba78bc9d791a3cf\": container with ID starting with 87e1fa104765b48dff286738580ba8ee8ac13b7f267b03ad3ba78bc9d791a3cf not found: ID does not exist"
Sep 19 11:59:21.494492 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:59:21.494467    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6447-03b5f2\" is forbidden: User \"system:node:ip-10-0-133-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-209.ec2.internal' and this object" podUID="8f8e260b-4048-4a7b-a405-0db54580cf2c" pod="opendatahub/test-curl-6447-03b5f2"
Sep 19 11:59:21.902908 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:21.902865    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f8e260b-4048-4a7b-a405-0db54580cf2c" path="/var/lib/kubelet/pods/8f8e260b-4048-4a7b-a405-0db54580cf2c/volumes"
Sep 19 11:59:22.523628 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:22.523586    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"]
Sep 19 11:59:22.540135 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:22.540082    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:59:22.651506 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:22.651459    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6447-5a4564"
Sep 19 11:59:22.739341 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:22.739249    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrbkl\" (UniqueName: \"kubernetes.io/projected/906ffcd6-d028-4c6a-9b80-0e27511df2c3-kube-api-access-nrbkl\") pod \"906ffcd6-d028-4c6a-9b80-0e27511df2c3\" (UID: \"906ffcd6-d028-4c6a-9b80-0e27511df2c3\") "
Sep 19 11:59:22.741537 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:22.741516    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/906ffcd6-d028-4c6a-9b80-0e27511df2c3-kube-api-access-nrbkl" (OuterVolumeSpecName: "kube-api-access-nrbkl") pod "906ffcd6-d028-4c6a-9b80-0e27511df2c3" (UID: "906ffcd6-d028-4c6a-9b80-0e27511df2c3"). InnerVolumeSpecName "kube-api-access-nrbkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:59:22.840284 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:22.840246    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrbkl\" (UniqueName: \"kubernetes.io/projected/906ffcd6-d028-4c6a-9b80-0e27511df2c3-kube-api-access-nrbkl\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:59:23.495678 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:23.495641    2584 generic.go:356] "Generic (PLEG): container finished" podID="906ffcd6-d028-4c6a-9b80-0e27511df2c3" containerID="d0a6393974eb13dfdc9def960bdb11f976e05f23b4f2a9a88efc910d682f3a95" exitCode=137
Sep 19 11:59:23.495678 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:23.495681    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6447-5a4564"
Sep 19 11:59:23.496002 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:23.495742    2584 scope.go:117] "RemoveContainer" containerID="d0a6393974eb13dfdc9def960bdb11f976e05f23b4f2a9a88efc910d682f3a95"
Sep 19 11:59:23.504668 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:23.504651    2584 scope.go:117] "RemoveContainer" containerID="d0a6393974eb13dfdc9def960bdb11f976e05f23b4f2a9a88efc910d682f3a95"
Sep 19 11:59:23.505022 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:59:23.505000    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0a6393974eb13dfdc9def960bdb11f976e05f23b4f2a9a88efc910d682f3a95\": container with ID starting with d0a6393974eb13dfdc9def960bdb11f976e05f23b4f2a9a88efc910d682f3a95 not found: ID does not exist" containerID="d0a6393974eb13dfdc9def960bdb11f976e05f23b4f2a9a88efc910d682f3a95"
Sep 19 11:59:23.505096 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:23.505033    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0a6393974eb13dfdc9def960bdb11f976e05f23b4f2a9a88efc910d682f3a95"} err="failed to get container status \"d0a6393974eb13dfdc9def960bdb11f976e05f23b4f2a9a88efc910d682f3a95\": rpc error: code = NotFound desc = could not find container \"d0a6393974eb13dfdc9def960bdb11f976e05f23b4f2a9a88efc910d682f3a95\": container with ID starting with d0a6393974eb13dfdc9def960bdb11f976e05f23b4f2a9a88efc910d682f3a95 not found: ID does not exist"
Sep 19 11:59:23.506359 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:59:23.506335    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6447-5a4564\" is forbidden: User \"system:node:ip-10-0-133-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-209.ec2.internal' and this object" podUID="906ffcd6-d028-4c6a-9b80-0e27511df2c3" pod="opendatahub/test-curl-6447-5a4564"
Sep 19 11:59:23.903478 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:23.903445    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="906ffcd6-d028-4c6a-9b80-0e27511df2c3" path="/var/lib/kubelet/pods/906ffcd6-d028-4c6a-9b80-0e27511df2c3/volumes"
Sep 19 11:59:27.040160 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:27.040128    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:59:27.096159 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:27.096104    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:59:36.161971 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.161901    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q" podUID="8cf5f279-6a4f-4470-b8d0-f9733d0d4a63" containerName="main" containerID="cri-o://42c1b2e4197abf69a013dd315b30d4058d4bdfedb5c8f336d65841e50113e5a6" gracePeriod=45
Sep 19 11:59:36.195785 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.195699    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q" podUID="8cf5f279-6a4f-4470-b8d0-f9733d0d4a63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 19 11:59:36.432783 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.432756    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:59:36.437885 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.437862    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl7k8\" (UniqueName: \"kubernetes.io/projected/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-kube-api-access-hl7k8\") pod \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") "
Sep 19 11:59:36.438035 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.437898    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-tmp-dir\") pod \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") "
Sep 19 11:59:36.438035 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.437940    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-model-cache\") pod \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") "
Sep 19 11:59:36.438035 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.437966    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-tls-certs\") pod \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") "
Sep 19 11:59:36.438035 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.438025    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-dshm\") pod \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") "
Sep 19 11:59:36.438245 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.438058    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-home\") pod \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\" (UID: \"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63\") "
Sep 19 11:59:36.438245 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.438211    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8cf5f279-6a4f-4470-b8d0-f9733d0d4a63" (UID: "8cf5f279-6a4f-4470-b8d0-f9733d0d4a63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:59:36.438245 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.438222    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-model-cache" (OuterVolumeSpecName: "model-cache") pod "8cf5f279-6a4f-4470-b8d0-f9733d0d4a63" (UID: "8cf5f279-6a4f-4470-b8d0-f9733d0d4a63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:59:36.438417 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.438394    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-home" (OuterVolumeSpecName: "home") pod "8cf5f279-6a4f-4470-b8d0-f9733d0d4a63" (UID: "8cf5f279-6a4f-4470-b8d0-f9733d0d4a63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:59:36.438485 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.438469    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-tmp-dir\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:59:36.438537 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.438489    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-model-cache\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:59:36.438537 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.438500    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-home\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:59:36.440350 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.440314    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-dshm" (OuterVolumeSpecName: "dshm") pod "8cf5f279-6a4f-4470-b8d0-f9733d0d4a63" (UID: "8cf5f279-6a4f-4470-b8d0-f9733d0d4a63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:59:36.440865 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.440840    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-kube-api-access-hl7k8" (OuterVolumeSpecName: "kube-api-access-hl7k8") pod "8cf5f279-6a4f-4470-b8d0-f9733d0d4a63" (UID: "8cf5f279-6a4f-4470-b8d0-f9733d0d4a63"). InnerVolumeSpecName "kube-api-access-hl7k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:59:36.440865 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.440844    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8cf5f279-6a4f-4470-b8d0-f9733d0d4a63" (UID: "8cf5f279-6a4f-4470-b8d0-f9733d0d4a63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:59:36.539047 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.539003    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-tls-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:59:36.539047 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.539036    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-dshm\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:59:36.539047 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.539047    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl7k8\" (UniqueName: \"kubernetes.io/projected/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63-kube-api-access-hl7k8\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:59:36.564467 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.564430    2584 generic.go:356] "Generic (PLEG): container finished" podID="8cf5f279-6a4f-4470-b8d0-f9733d0d4a63" containerID="42c1b2e4197abf69a013dd315b30d4058d4bdfedb5c8f336d65841e50113e5a6" exitCode=0
Sep 19 11:59:36.564680 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.564487    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"
Sep 19 11:59:36.564680 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.564517    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q" event={"ID":"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63","Type":"ContainerDied","Data":"42c1b2e4197abf69a013dd315b30d4058d4bdfedb5c8f336d65841e50113e5a6"}
Sep 19 11:59:36.564680 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.564570    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q" event={"ID":"8cf5f279-6a4f-4470-b8d0-f9733d0d4a63","Type":"ContainerDied","Data":"6752a3f05e868b721bd600b95212ddd35b13faae23624bc0e63ef91cdf4237bb"}
Sep 19 11:59:36.564680 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.564592    2584 scope.go:117] "RemoveContainer" containerID="42c1b2e4197abf69a013dd315b30d4058d4bdfedb5c8f336d65841e50113e5a6"
Sep 19 11:59:36.574949 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.574892    2584 scope.go:117] "RemoveContainer" containerID="42c1b2e4197abf69a013dd315b30d4058d4bdfedb5c8f336d65841e50113e5a6"
Sep 19 11:59:36.575293 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:59:36.575262    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c1b2e4197abf69a013dd315b30d4058d4bdfedb5c8f336d65841e50113e5a6\": container with ID starting with 42c1b2e4197abf69a013dd315b30d4058d4bdfedb5c8f336d65841e50113e5a6 not found: ID does not exist" containerID="42c1b2e4197abf69a013dd315b30d4058d4bdfedb5c8f336d65841e50113e5a6"
Sep 19 11:59:36.575363 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.575303    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c1b2e4197abf69a013dd315b30d4058d4bdfedb5c8f336d65841e50113e5a6"} err="failed to get container status \"42c1b2e4197abf69a013dd315b30d4058d4bdfedb5c8f336d65841e50113e5a6\": rpc error: code = NotFound desc = could not find container \"42c1b2e4197abf69a013dd315b30d4058d4bdfedb5c8f336d65841e50113e5a6\": container with ID starting with 42c1b2e4197abf69a013dd315b30d4058d4bdfedb5c8f336d65841e50113e5a6 not found: ID does not exist"
Sep 19 11:59:36.587364 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.587339    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"]
Sep 19 11:59:36.590884 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:36.590854    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4937ff/sub-test-model-4937ff-kserve-7b7b54bd46-pjj9q"]
Sep 19 11:59:37.545560 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.545517    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd" podUID="e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff" containerName="main" containerID="cri-o://7e295d47a18db3a6c861896e10591d06db159bbe8dddef722525055a2474f5b8" gracePeriod=45
Sep 19 11:59:37.808666 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.808593    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:59:37.851190 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.851154    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-tls-certs\") pod \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") "
Sep 19 11:59:37.851358 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.851230    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-tmp-dir\") pod \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") "
Sep 19 11:59:37.851358 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.851255    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s58gv\" (UniqueName: \"kubernetes.io/projected/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-kube-api-access-s58gv\") pod \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") "
Sep 19 11:59:37.851358 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.851272    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-dshm\") pod \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") "
Sep 19 11:59:37.851358 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.851305    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-model-cache\") pod \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") "
Sep 19 11:59:37.851358 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.851356    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-home\") pod \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\" (UID: \"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff\") "
Sep 19 11:59:37.851586 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.851542    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff" (UID: "e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:59:37.851657 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.851616    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff" (UID: "e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:59:37.851657 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.851645    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-tmp-dir\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:59:37.851766 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.851709    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-home" (OuterVolumeSpecName: "home") pod "e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff" (UID: "e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:59:37.853556 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.853533    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff" (UID: "e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 11:59:37.853642 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.853613    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-dshm" (OuterVolumeSpecName: "dshm") pod "e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff" (UID: "e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 11:59:37.853642 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.853630    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-kube-api-access-s58gv" (OuterVolumeSpecName: "kube-api-access-s58gv") pod "e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff" (UID: "e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff"). InnerVolumeSpecName "kube-api-access-s58gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 11:59:37.903572 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.903541    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cf5f279-6a4f-4470-b8d0-f9733d0d4a63" path="/var/lib/kubelet/pods/8cf5f279-6a4f-4470-b8d0-f9733d0d4a63/volumes"
Sep 19 11:59:37.952762 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.952658    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s58gv\" (UniqueName: \"kubernetes.io/projected/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-kube-api-access-s58gv\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:59:37.952762 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.952692    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-dshm\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:59:37.952762 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.952712    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-model-cache\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:59:37.952762 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.952725    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-home\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:59:37.952762 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:37.952738    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff-tls-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 11:59:38.577435 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:38.577396    2584 generic.go:356] "Generic (PLEG): container finished" podID="e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff" containerID="7e295d47a18db3a6c861896e10591d06db159bbe8dddef722525055a2474f5b8" exitCode=0
Sep 19 11:59:38.577874 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:38.577448    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"
Sep 19 11:59:38.577874 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:38.577447    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd" event={"ID":"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff","Type":"ContainerDied","Data":"7e295d47a18db3a6c861896e10591d06db159bbe8dddef722525055a2474f5b8"}
Sep 19 11:59:38.577874 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:38.577487    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd" event={"ID":"e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff","Type":"ContainerDied","Data":"6e7c47ed786e2546d1d95ff4ae5175a399a29ed5865e5ae807876acc1a77bd19"}
Sep 19 11:59:38.577874 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:38.577511    2584 scope.go:117] "RemoveContainer" containerID="7e295d47a18db3a6c861896e10591d06db159bbe8dddef722525055a2474f5b8"
Sep 19 11:59:38.586382 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:38.586359    2584 scope.go:117] "RemoveContainer" containerID="7e295d47a18db3a6c861896e10591d06db159bbe8dddef722525055a2474f5b8"
Sep 19 11:59:38.586710 ip-10-0-133-209 kubenswrapper[2584]: E0919 11:59:38.586694    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e295d47a18db3a6c861896e10591d06db159bbe8dddef722525055a2474f5b8\": container with ID starting with 7e295d47a18db3a6c861896e10591d06db159bbe8dddef722525055a2474f5b8 not found: ID does not exist" containerID="7e295d47a18db3a6c861896e10591d06db159bbe8dddef722525055a2474f5b8"
Sep 19 11:59:38.586769 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:38.586719    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e295d47a18db3a6c861896e10591d06db159bbe8dddef722525055a2474f5b8"} err="failed to get container status \"7e295d47a18db3a6c861896e10591d06db159bbe8dddef722525055a2474f5b8\": rpc error: code = NotFound desc = could not find container \"7e295d47a18db3a6c861896e10591d06db159bbe8dddef722525055a2474f5b8\": container with ID starting with 7e295d47a18db3a6c861896e10591d06db159bbe8dddef722525055a2474f5b8 not found: ID does not exist"
Sep 19 11:59:38.593304 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:38.593274    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"]
Sep 19 11:59:38.601551 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:38.599749    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a8a59/sub-test-model-4a8a59-kserve-c4977b856-8npfd"]
Sep 19 11:59:39.903459 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:39.903417    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff" path="/var/lib/kubelet/pods/e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff/volumes"
Sep 19 11:59:45.024406 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.024375    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"]
Sep 19 11:59:45.024779 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.024722    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff" containerName="main"
Sep 19 11:59:45.024779 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.024733    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff" containerName="main"
Sep 19 11:59:45.024779 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.024742    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="906ffcd6-d028-4c6a-9b80-0e27511df2c3" containerName="test-curl-6447-5a4564"
Sep 19 11:59:45.024779 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.024749    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="906ffcd6-d028-4c6a-9b80-0e27511df2c3" containerName="test-curl-6447-5a4564"
Sep 19 11:59:45.024779 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.024770    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cf5f279-6a4f-4470-b8d0-f9733d0d4a63" containerName="main"
Sep 19 11:59:45.024779 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.024776    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cf5f279-6a4f-4470-b8d0-f9733d0d4a63" containerName="main"
Sep 19 11:59:45.024982 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.024831    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cf5f279-6a4f-4470-b8d0-f9733d0d4a63" containerName="main"
Sep 19 11:59:45.024982 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.024841    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8d259e5-9aa8-4e6d-aa26-1bda21f9d6ff" containerName="main"
Sep 19 11:59:45.024982 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.024848    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="906ffcd6-d028-4c6a-9b80-0e27511df2c3" containerName="test-curl-6447-5a4564"
Sep 19 11:59:45.029236 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.029216    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.040730 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.040695    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4937ff\"/\"kube-root-ca.crt\""
Sep 19 11:59:45.040955 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.040810    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4937ff\"/\"rate-test-model-4937ff-kserve-self-signed-certs\""
Sep 19 11:59:45.040955 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.040828    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4937ff\"/\"openshift-service-ca.crt\""
Sep 19 11:59:45.041494 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.041469    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4937ff\"/\"default-dockercfg-7njzd\""
Sep 19 11:59:45.044553 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.044515    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"]
Sep 19 11:59:45.055218 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.055180    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 11:59:45.116372 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.116335    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-model-cache\") pod \"rate-test-model-4937ff-kserve-8459b77dfb-4rrg2\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.116578 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.116399    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-dshm\") pod \"rate-test-model-4937ff-kserve-8459b77dfb-4rrg2\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.116578 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.116464    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9pkg\" (UniqueName: \"kubernetes.io/projected/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-kube-api-access-n9pkg\") pod \"rate-test-model-4937ff-kserve-8459b77dfb-4rrg2\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.116578 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.116495    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-home\") pod \"rate-test-model-4937ff-kserve-8459b77dfb-4rrg2\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.116578 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.116515    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-tls-certs\") pod \"rate-test-model-4937ff-kserve-8459b77dfb-4rrg2\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.116578 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.116577    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-tmp-dir\") pod \"rate-test-model-4937ff-kserve-8459b77dfb-4rrg2\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.217765 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.217722    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-dshm\") pod \"rate-test-model-4937ff-kserve-8459b77dfb-4rrg2\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.217997 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.217775    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9pkg\" (UniqueName: \"kubernetes.io/projected/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-kube-api-access-n9pkg\") pod \"rate-test-model-4937ff-kserve-8459b77dfb-4rrg2\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.217997 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.217899    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-home\") pod \"rate-test-model-4937ff-kserve-8459b77dfb-4rrg2\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.217997 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.217968    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-tls-certs\") pod \"rate-test-model-4937ff-kserve-8459b77dfb-4rrg2\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.218204 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.218032    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-tmp-dir\") pod \"rate-test-model-4937ff-kserve-8459b77dfb-4rrg2\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.218204 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.218110    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-model-cache\") pod \"rate-test-model-4937ff-kserve-8459b77dfb-4rrg2\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.218346 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.218314    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-home\") pod \"rate-test-model-4937ff-kserve-8459b77dfb-4rrg2\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.218456 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.218433    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-tmp-dir\") pod \"rate-test-model-4937ff-kserve-8459b77dfb-4rrg2\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.218500 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.218471    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-model-cache\") pod \"rate-test-model-4937ff-kserve-8459b77dfb-4rrg2\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.220217 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.220200    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-dshm\") pod \"rate-test-model-4937ff-kserve-8459b77dfb-4rrg2\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.220547 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.220529    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-tls-certs\") pod \"rate-test-model-4937ff-kserve-8459b77dfb-4rrg2\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.225855 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.225832    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9pkg\" (UniqueName: \"kubernetes.io/projected/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-kube-api-access-n9pkg\") pod \"rate-test-model-4937ff-kserve-8459b77dfb-4rrg2\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") " pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.348571 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.348535    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:45.623378 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:45.623351    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"]
Sep 19 11:59:45.625579 ip-10-0-133-209 kubenswrapper[2584]: W0919 11:59:45.625551    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad8f4c73_cf9b_4168_907f_6ef8bfd7a23d.slice/crio-9d98dfa4cb888dfac29ec97a1cd21f5f9f0901d92d72c60e833462ef1b1afa4e WatchSource:0}: Error finding container 9d98dfa4cb888dfac29ec97a1cd21f5f9f0901d92d72c60e833462ef1b1afa4e: Status 404 returned error can't find the container with id 9d98dfa4cb888dfac29ec97a1cd21f5f9f0901d92d72c60e833462ef1b1afa4e
Sep 19 11:59:46.622454 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:46.622419    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2" event={"ID":"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d","Type":"ContainerStarted","Data":"df3435af8740922a9770e0c1b153ead6aad28acc6d962bf164220041256cbb1b"}
Sep 19 11:59:46.622934 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:46.622467    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2" event={"ID":"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d","Type":"ContainerStarted","Data":"9d98dfa4cb888dfac29ec97a1cd21f5f9f0901d92d72c60e833462ef1b1afa4e"}
Sep 19 11:59:46.638481 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:46.638425    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2" podStartSLOduration=1.6384099989999998 podStartE2EDuration="1.638409999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 11:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 11:59:46.637112714 +0000 UTC m=+1570.837142517" watchObservedRunningTime="2026-09-19 11:59:46.638409999 +0000 UTC m=+1570.838439802"
Sep 19 11:59:47.349023 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:47.348989    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:55.348830 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:55.348788    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:55.361563 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:55.361530    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 11:59:55.678901 ip-10-0-133-209 kubenswrapper[2584]: I0919 11:59:55.678797    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 12:00:00.131261 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:00.131223    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4937ff-29830320-svghl"]
Sep 19 12:00:00.134557 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:00.134533    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4937ff-29830320-svghl"
Sep 19 12:00:00.136905 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:00.136882    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bnlmc\""
Sep 19 12:00:00.141096 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:00.141066    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4937ff-29830320-svghl"]
Sep 19 12:00:00.148005 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:00.147978    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4937ff-29830320-svghl"
Sep 19 12:00:00.421495 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:00.421469    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4937ff-29830320-svghl"]
Sep 19 12:00:00.424096 ip-10-0-133-209 kubenswrapper[2584]: W0919 12:00:00.424070    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e583e2a_2f1f_41a6_bbd8_2430b405bb3d.slice/crio-c109b22076bb723db5650b6c26ac6c32b71d558aebfbbaae96fd35bbe4c015c7 WatchSource:0}: Error finding container c109b22076bb723db5650b6c26ac6c32b71d558aebfbbaae96fd35bbe4c015c7: Status 404 returned error can't find the container with id c109b22076bb723db5650b6c26ac6c32b71d558aebfbbaae96fd35bbe4c015c7
Sep 19 12:00:00.426377 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:00.426357    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 12:00:00.694136 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:00.694044    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4937ff-29830320-svghl" event={"ID":"1e583e2a-2f1f-41a6-bbd8-2430b405bb3d","Type":"ContainerStarted","Data":"c109b22076bb723db5650b6c26ac6c32b71d558aebfbbaae96fd35bbe4c015c7"}
Sep 19 12:00:03.713722 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:03.713687    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4937ff-29830320-svghl" event={"ID":"1e583e2a-2f1f-41a6-bbd8-2430b405bb3d","Type":"ContainerStarted","Data":"478781711d006fcc1615d0bbadbe0c82fa04768592b324c882227db11347a419"}
Sep 19 12:00:03.730783 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:03.730725    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4937ff-29830320-svghl" podStartSLOduration=1.26189954 podStartE2EDuration="3.730709391s" totalImagesPullingTime="2.468809851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:00:03.728882727 +0000 UTC m=+1587.928912529" watchObservedRunningTime="2026-09-19 12:00:03.730709391 +0000 UTC m=+1587.930739193"
Sep 19 12:00:04.719862 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:04.719826    2584 generic.go:356] "Generic (PLEG): container finished" podID="1e583e2a-2f1f-41a6-bbd8-2430b405bb3d" containerID="478781711d006fcc1615d0bbadbe0c82fa04768592b324c882227db11347a419" exitCode=0
Sep 19 12:00:04.720344 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:04.719907    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4937ff-29830320-svghl" event={"ID":"1e583e2a-2f1f-41a6-bbd8-2430b405bb3d","Type":"ContainerDied","Data":"478781711d006fcc1615d0bbadbe0c82fa04768592b324c882227db11347a419"}
Sep 19 12:00:05.868748 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:05.868711    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4937ff-29830320-svghl"
Sep 19 12:00:06.731441 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:06.731412    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4937ff-29830320-svghl"
Sep 19 12:00:06.731637 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:06.731409    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4937ff-29830320-svghl" event={"ID":"1e583e2a-2f1f-41a6-bbd8-2430b405bb3d","Type":"ContainerDied","Data":"c109b22076bb723db5650b6c26ac6c32b71d558aebfbbaae96fd35bbe4c015c7"}
Sep 19 12:00:06.731637 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:06.731534    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c109b22076bb723db5650b6c26ac6c32b71d558aebfbbaae96fd35bbe4c015c7"
Sep 19 12:00:09.811641 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:09.811607    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:00:12.725048 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.725012    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"]
Sep 19 12:00:12.725478 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.725399    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e583e2a-2f1f-41a6-bbd8-2430b405bb3d" containerName="cleanup"
Sep 19 12:00:12.725478 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.725410    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e583e2a-2f1f-41a6-bbd8-2430b405bb3d" containerName="cleanup"
Sep 19 12:00:12.725478 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.725479    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e583e2a-2f1f-41a6-bbd8-2430b405bb3d" containerName="cleanup"
Sep 19 12:00:12.728568 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.728547    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:12.731338 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.731308    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4a8a59\"/\"kube-root-ca.crt\""
Sep 19 12:00:12.731479 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.731322    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4a8a59\"/\"openshift-service-ca.crt\""
Sep 19 12:00:12.731479 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.731352    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4a8a59\"/\"rate-test-model-4a8a59-kserve-self-signed-certs\""
Sep 19 12:00:12.731479 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.731446    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4a8a59\"/\"default-dockercfg-7wptg\""
Sep 19 12:00:12.736541 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.736256    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"]
Sep 19 12:00:12.749558 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.749523    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:00:12.884412 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.884373    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-tmp-dir\") pod \"rate-test-model-4a8a59-kserve-664475b648-dd5ml\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:12.884614 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.884428    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-home\") pod \"rate-test-model-4a8a59-kserve-664475b648-dd5ml\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:12.884614 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.884490    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5dxh\" (UniqueName: \"kubernetes.io/projected/39505ce3-63b9-4972-841f-42a92b4ad14d-kube-api-access-l5dxh\") pod \"rate-test-model-4a8a59-kserve-664475b648-dd5ml\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:12.884614 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.884509    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39505ce3-63b9-4972-841f-42a92b4ad14d-tls-certs\") pod \"rate-test-model-4a8a59-kserve-664475b648-dd5ml\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:12.884614 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.884531    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-model-cache\") pod \"rate-test-model-4a8a59-kserve-664475b648-dd5ml\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:12.884614 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.884569    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-dshm\") pod \"rate-test-model-4a8a59-kserve-664475b648-dd5ml\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:12.985168 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.985056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5dxh\" (UniqueName: \"kubernetes.io/projected/39505ce3-63b9-4972-841f-42a92b4ad14d-kube-api-access-l5dxh\") pod \"rate-test-model-4a8a59-kserve-664475b648-dd5ml\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:12.985168 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.985114    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39505ce3-63b9-4972-841f-42a92b4ad14d-tls-certs\") pod \"rate-test-model-4a8a59-kserve-664475b648-dd5ml\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:12.985168 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.985148    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-model-cache\") pod \"rate-test-model-4a8a59-kserve-664475b648-dd5ml\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:12.985512 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.985206    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-dshm\") pod \"rate-test-model-4a8a59-kserve-664475b648-dd5ml\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:12.985512 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.985236    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-tmp-dir\") pod \"rate-test-model-4a8a59-kserve-664475b648-dd5ml\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:12.985512 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.985283    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-home\") pod \"rate-test-model-4a8a59-kserve-664475b648-dd5ml\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:12.985676 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.985596    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-model-cache\") pod \"rate-test-model-4a8a59-kserve-664475b648-dd5ml\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:12.985741 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.985729    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-home\") pod \"rate-test-model-4a8a59-kserve-664475b648-dd5ml\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:12.985825 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.985801    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-tmp-dir\") pod \"rate-test-model-4a8a59-kserve-664475b648-dd5ml\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:12.988315 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.988290    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-dshm\") pod \"rate-test-model-4a8a59-kserve-664475b648-dd5ml\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:12.988664 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.988644    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39505ce3-63b9-4972-841f-42a92b4ad14d-tls-certs\") pod \"rate-test-model-4a8a59-kserve-664475b648-dd5ml\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:12.993466 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:12.993438    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5dxh\" (UniqueName: \"kubernetes.io/projected/39505ce3-63b9-4972-841f-42a92b4ad14d-kube-api-access-l5dxh\") pod \"rate-test-model-4a8a59-kserve-664475b648-dd5ml\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") " pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:13.039543 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:13.039503    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:13.315300 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:13.315266    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"]
Sep 19 12:00:13.318196 ip-10-0-133-209 kubenswrapper[2584]: W0919 12:00:13.318164    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39505ce3_63b9_4972_841f_42a92b4ad14d.slice/crio-082deedbf21c2f8a31a012bb78f6ca9b8e038b8453b3adbbe9a663b358fad222 WatchSource:0}: Error finding container 082deedbf21c2f8a31a012bb78f6ca9b8e038b8453b3adbbe9a663b358fad222: Status 404 returned error can't find the container with id 082deedbf21c2f8a31a012bb78f6ca9b8e038b8453b3adbbe9a663b358fad222
Sep 19 12:00:13.774534 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:13.774429    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml" event={"ID":"39505ce3-63b9-4972-841f-42a92b4ad14d","Type":"ContainerStarted","Data":"a7ee873b626ac9f40b078bc44588e5526df0c9c468e40e2ec6bb13b20195c5ce"}
Sep 19 12:00:13.774534 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:13.774475    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml" event={"ID":"39505ce3-63b9-4972-841f-42a92b4ad14d","Type":"ContainerStarted","Data":"082deedbf21c2f8a31a012bb78f6ca9b8e038b8453b3adbbe9a663b358fad222"}
Sep 19 12:00:13.793413 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:13.793352    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml" podStartSLOduration=1.793333166 podStartE2EDuration="1.793333166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:00:13.790319088 +0000 UTC m=+1597.990348911" watchObservedRunningTime="2026-09-19 12:00:13.793333166 +0000 UTC m=+1597.993362969"
Sep 19 12:00:14.039729 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:14.039635    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:23.040307 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:23.040254    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:23.053646 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:23.053615    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:00:23.836760 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:00:23.836728    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:01:00.638192 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:00.638155    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"]
Sep 19 12:01:00.657532 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:00.657500    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:02.155776 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:02.155742    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"]
Sep 19 12:01:02.179259 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:02.179228    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:02.603029 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:02.602994    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:02.886693 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:02.886593    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"]
Sep 19 12:01:02.890128 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:02.890098    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:02.892856 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:02.892832    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 19 12:01:02.893039 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:02.892891    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-8a618a-openshift-default-dockercfg-cbv6b\""
Sep 19 12:01:02.902165 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:02.902134    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"]
Sep 19 12:01:02.910992 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:02.910955    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:02.960007 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:02.959958    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82514835-6ded-4b36-96b1-6e260624de3f-istio-token\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:02.960212 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:02.960029    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82514835-6ded-4b36-96b1-6e260624de3f-istiod-ca-cert\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:02.960212 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:02.960072    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-workload-socket\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:02.960212 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:02.960090    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgfqd\" (UniqueName: \"kubernetes.io/projected/82514835-6ded-4b36-96b1-6e260624de3f-kube-api-access-kgfqd\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:02.960212 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:02.960131    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82514835-6ded-4b36-96b1-6e260624de3f-istio-podinfo\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:02.960212 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:02.960160    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-istio-data\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:02.960438 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:02.960223    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-credential-socket\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:02.960438 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:02.960310    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-istio-envoy\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:02.960438 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:02.960335    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-workload-certs\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.061589 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.061546    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-istio-envoy\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.061589 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.061592    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-workload-certs\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.061866 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.061628    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82514835-6ded-4b36-96b1-6e260624de3f-istio-token\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.061866 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.061657    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82514835-6ded-4b36-96b1-6e260624de3f-istiod-ca-cert\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.061866 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.061688    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-workload-socket\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.061866 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.061711    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgfqd\" (UniqueName: \"kubernetes.io/projected/82514835-6ded-4b36-96b1-6e260624de3f-kube-api-access-kgfqd\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.061866 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.061753    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82514835-6ded-4b36-96b1-6e260624de3f-istio-podinfo\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.061866 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.061783    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-istio-data\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.061866 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.061814    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-credential-socket\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.062266 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.062160    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-workload-certs\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.062266 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.062195    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-workload-socket\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.062463 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.062436    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-credential-socket\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.062769 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.062479    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-istio-data\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.062769 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.062672    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82514835-6ded-4b36-96b1-6e260624de3f-istiod-ca-cert\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.064176 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.064154    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-istio-envoy\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.064403 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.064384    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82514835-6ded-4b36-96b1-6e260624de3f-istio-podinfo\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.070022 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.069997    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82514835-6ded-4b36-96b1-6e260624de3f-istio-token\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.070119 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.070070    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgfqd\" (UniqueName: \"kubernetes.io/projected/82514835-6ded-4b36-96b1-6e260624de3f-kube-api-access-kgfqd\") pod \"e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") " pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.161758 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.161672    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:03.203639 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.203595    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:03.483198 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.483172    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"]
Sep 19 12:01:03.485355 ip-10-0-133-209 kubenswrapper[2584]: W0919 12:01:03.485325    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82514835_6ded_4b36_96b1_6e260624de3f.slice/crio-dbdb19ce7dbe4fbc659cfaf2ccb339b907f4314070d849d72dadef676a1cf5b3 WatchSource:0}: Error finding container dbdb19ce7dbe4fbc659cfaf2ccb339b907f4314070d849d72dadef676a1cf5b3: Status 404 returned error can't find the container with id dbdb19ce7dbe4fbc659cfaf2ccb339b907f4314070d849d72dadef676a1cf5b3
Sep 19 12:01:03.487737 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.487701    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 12:01:03.487835 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.487785    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 12:01:03.487902 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:03.487832    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 12:01:04.031627 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:04.031587    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw" event={"ID":"82514835-6ded-4b36-96b1-6e260624de3f","Type":"ContainerStarted","Data":"c80e9979d2e67463f66c5eaa1866d33ab5719d9e6f16b7fdf7a2e14789268805"}
Sep 19 12:01:04.031818 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:04.031654    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw" event={"ID":"82514835-6ded-4b36-96b1-6e260624de3f","Type":"ContainerStarted","Data":"dbdb19ce7dbe4fbc659cfaf2ccb339b907f4314070d849d72dadef676a1cf5b3"}
Sep 19 12:01:04.052326 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:04.052274    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw" podStartSLOduration=2.052257854 podStartE2EDuration="2.052257854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:01:04.05096513 +0000 UTC m=+1648.250994934" watchObservedRunningTime="2026-09-19 12:01:04.052257854 +0000 UTC m=+1648.252287657"
Sep 19 12:01:04.204631 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:04.204587    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:04.209803 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:04.209777    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:05.037211 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:05.037180    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:05.038303 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:05.038282    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:05.132019 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:05.131974    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:05.957848 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:05.957811    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:06.052328 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:06.052297    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:06.716516 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:06.716483    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:07.438952 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:07.438900    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"]
Sep 19 12:01:07.442492 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:07.442472    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"
Sep 19 12:01:07.445034 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:07.445011    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8a618a-serving-cert\""
Sep 19 12:01:07.452499 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:07.452472    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"]
Sep 19 12:01:07.466324 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:07.466294    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:07.506141 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:07.506108    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49132f35-53b0-42fb-977c-f0566ce60304-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg\" (UID: \"49132f35-53b0-42fb-977c-f0566ce60304\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"
Sep 19 12:01:07.506357 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:07.506166    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ncgz\" (UniqueName: \"kubernetes.io/projected/49132f35-53b0-42fb-977c-f0566ce60304-kube-api-access-6ncgz\") pod \"maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg\" (UID: \"49132f35-53b0-42fb-977c-f0566ce60304\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"
Sep 19 12:01:07.506357 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:07.506294    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49132f35-53b0-42fb-977c-f0566ce60304-metrics-certs\") pod \"maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg\" (UID: \"49132f35-53b0-42fb-977c-f0566ce60304\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"
Sep 19 12:01:07.506479 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:07.506362    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49132f35-53b0-42fb-977c-f0566ce60304-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg\" (UID: \"49132f35-53b0-42fb-977c-f0566ce60304\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"
Sep 19 12:01:07.562695 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:07.562657    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:07.607830 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:07.607788    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49132f35-53b0-42fb-977c-f0566ce60304-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg\" (UID: \"49132f35-53b0-42fb-977c-f0566ce60304\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"
Sep 19 12:01:07.608069 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:07.607854    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ncgz\" (UniqueName: \"kubernetes.io/projected/49132f35-53b0-42fb-977c-f0566ce60304-kube-api-access-6ncgz\") pod \"maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg\" (UID: \"49132f35-53b0-42fb-977c-f0566ce60304\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"
Sep 19 12:01:07.608069 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:07.607901    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49132f35-53b0-42fb-977c-f0566ce60304-metrics-certs\") pod \"maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg\" (UID: \"49132f35-53b0-42fb-977c-f0566ce60304\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"
Sep 19 12:01:07.608069 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:07.607966    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49132f35-53b0-42fb-977c-f0566ce60304-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg\" (UID: \"49132f35-53b0-42fb-977c-f0566ce60304\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"
Sep 19 12:01:07.608316 ip-10-0-133-209 kubenswrapper[2584]: E0919 12:01:07.608076    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-serving-cert: secret "maas-api-e2e-ipp-a-8a618a-serving-cert" not found
Sep 19 12:01:07.608316 ip-10-0-133-209 kubenswrapper[2584]: E0919 12:01:07.608132    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49132f35-53b0-42fb-977c-f0566ce60304-maas-api-tls podName:49132f35-53b0-42fb-977c-f0566ce60304 nodeName:}" failed. No retries permitted until 2026-09-19 12:01:08.108109817 +0000 UTC m=+1652.308139611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/49132f35-53b0-42fb-977c-f0566ce60304-maas-api-tls") pod "maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg" (UID: "49132f35-53b0-42fb-977c-f0566ce60304") : secret "maas-api-e2e-ipp-a-8a618a-serving-cert" not found
Sep 19 12:01:07.608637 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:07.608610    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49132f35-53b0-42fb-977c-f0566ce60304-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg\" (UID: \"49132f35-53b0-42fb-977c-f0566ce60304\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"
Sep 19 12:01:07.610593 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:07.610572    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49132f35-53b0-42fb-977c-f0566ce60304-metrics-certs\") pod \"maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg\" (UID: \"49132f35-53b0-42fb-977c-f0566ce60304\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"
Sep 19 12:01:07.617110 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:07.617063    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ncgz\" (UniqueName: \"kubernetes.io/projected/49132f35-53b0-42fb-977c-f0566ce60304-kube-api-access-6ncgz\") pod \"maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg\" (UID: \"49132f35-53b0-42fb-977c-f0566ce60304\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"
Sep 19 12:01:07.854839 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:07.854803    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:07.954897 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:07.954863    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:08.026517 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:08.026478    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:08.113113 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:08.113002    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49132f35-53b0-42fb-977c-f0566ce60304-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg\" (UID: \"49132f35-53b0-42fb-977c-f0566ce60304\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"
Sep 19 12:01:08.115838 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:08.115775    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49132f35-53b0-42fb-977c-f0566ce60304-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg\" (UID: \"49132f35-53b0-42fb-977c-f0566ce60304\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"
Sep 19 12:01:08.352196 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:08.352161    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:08.356593 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:08.356559    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"
Sep 19 12:01:08.639324 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:08.638210    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"]
Sep 19 12:01:08.640013 ip-10-0-133-209 kubenswrapper[2584]: W0919 12:01:08.639987    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49132f35_53b0_42fb_977c_f0566ce60304.slice/crio-dc12b9592e5ad2001f4dac351e7e94b94e659a0347a68ae1992cb84b33e1097e WatchSource:0}: Error finding container dc12b9592e5ad2001f4dac351e7e94b94e659a0347a68ae1992cb84b33e1097e: Status 404 returned error can't find the container with id dc12b9592e5ad2001f4dac351e7e94b94e659a0347a68ae1992cb84b33e1097e
Sep 19 12:01:09.060664 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:09.060615    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg" event={"ID":"49132f35-53b0-42fb-977c-f0566ce60304","Type":"ContainerStarted","Data":"dc12b9592e5ad2001f4dac351e7e94b94e659a0347a68ae1992cb84b33e1097e"}
Sep 19 12:01:09.148643 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:09.148619    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:10.067448 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:10.067408    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg" event={"ID":"49132f35-53b0-42fb-977c-f0566ce60304","Type":"ContainerStarted","Data":"b44d7611f959c9777453b1aea8096b029dbe6ab8eb16559e829eaca6c089493e"}
Sep 19 12:01:10.067995 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:10.067543    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"
Sep 19 12:01:10.083973 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:10.083896    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg" podStartSLOduration=2.636367958 podStartE2EDuration="3.083878186s" totalImagesPullingTime="447.510228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:01:10.082704341 +0000 UTC m=+1654.282734147" watchObservedRunningTime="2026-09-19 12:01:10.083878186 +0000 UTC m=+1654.283908048"
Sep 19 12:01:15.658419 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:15.658352    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2" podUID="ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d" containerName="main" containerID="cri-o://df3435af8740922a9770e0c1b153ead6aad28acc6d962bf164220041256cbb1b" gracePeriod=45
Sep 19 12:01:15.668229 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:15.668189    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2" podUID="ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 19 12:01:15.922734 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:15.922664    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 12:01:15.993654 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:15.993617    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-tmp-dir\") pod \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") "
Sep 19 12:01:15.993654 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:15.993661    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-dshm\") pod \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") "
Sep 19 12:01:15.993953 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:15.993682    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-home\") pod \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") "
Sep 19 12:01:15.993953 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:15.993714    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-tls-certs\") pod \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") "
Sep 19 12:01:15.993953 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:15.993763    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-model-cache\") pod \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") "
Sep 19 12:01:15.994130 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:15.993942    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9pkg\" (UniqueName: \"kubernetes.io/projected/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-kube-api-access-n9pkg\") pod \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\" (UID: \"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d\") "
Sep 19 12:01:15.994130 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:15.994010    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-home" (OuterVolumeSpecName: "home") pod "ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d" (UID: "ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:01:15.994130 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:15.994022    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d" (UID: "ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:01:15.994264 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:15.994138    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d" (UID: "ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:01:15.994365 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:15.994342    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-tmp-dir\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:01:15.994432 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:15.994371    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-home\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:01:15.994432 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:15.994385    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-model-cache\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:01:15.996182 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:15.996157    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-dshm" (OuterVolumeSpecName: "dshm") pod "ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d" (UID: "ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:01:15.996687 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:15.996665    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-kube-api-access-n9pkg" (OuterVolumeSpecName: "kube-api-access-n9pkg") pod "ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d" (UID: "ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d"). InnerVolumeSpecName "kube-api-access-n9pkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 12:01:15.996687 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:15.996666    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d" (UID: "ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 12:01:16.078214 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:16.078185    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"
Sep 19 12:01:16.095746 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:16.095714    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-tls-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:01:16.095746 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:16.095749    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9pkg\" (UniqueName: \"kubernetes.io/projected/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-kube-api-access-n9pkg\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:01:16.096017 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:16.095763    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d-dshm\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:01:16.102145 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:16.102114    2584 generic.go:356] "Generic (PLEG): container finished" podID="ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d" containerID="df3435af8740922a9770e0c1b153ead6aad28acc6d962bf164220041256cbb1b" exitCode=0
Sep 19 12:01:16.102339 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:16.102164    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"
Sep 19 12:01:16.102339 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:16.102199    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2" event={"ID":"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d","Type":"ContainerDied","Data":"df3435af8740922a9770e0c1b153ead6aad28acc6d962bf164220041256cbb1b"}
Sep 19 12:01:16.102339 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:16.102245    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2" event={"ID":"ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d","Type":"ContainerDied","Data":"9d98dfa4cb888dfac29ec97a1cd21f5f9f0901d92d72c60e833462ef1b1afa4e"}
Sep 19 12:01:16.102339 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:16.102258    2584 scope.go:117] "RemoveContainer" containerID="df3435af8740922a9770e0c1b153ead6aad28acc6d962bf164220041256cbb1b"
Sep 19 12:01:16.118795 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:16.118770    2584 scope.go:117] "RemoveContainer" containerID="df3435af8740922a9770e0c1b153ead6aad28acc6d962bf164220041256cbb1b"
Sep 19 12:01:16.119196 ip-10-0-133-209 kubenswrapper[2584]: E0919 12:01:16.119173    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3435af8740922a9770e0c1b153ead6aad28acc6d962bf164220041256cbb1b\": container with ID starting with df3435af8740922a9770e0c1b153ead6aad28acc6d962bf164220041256cbb1b not found: ID does not exist" containerID="df3435af8740922a9770e0c1b153ead6aad28acc6d962bf164220041256cbb1b"
Sep 19 12:01:16.119293 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:16.119206    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3435af8740922a9770e0c1b153ead6aad28acc6d962bf164220041256cbb1b"} err="failed to get container status \"df3435af8740922a9770e0c1b153ead6aad28acc6d962bf164220041256cbb1b\": rpc error: code = NotFound desc = could not find container \"df3435af8740922a9770e0c1b153ead6aad28acc6d962bf164220041256cbb1b\": container with ID starting with df3435af8740922a9770e0c1b153ead6aad28acc6d962bf164220041256cbb1b not found: ID does not exist"
Sep 19 12:01:16.130973 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:16.130887    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"]
Sep 19 12:01:16.134459 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:16.134424    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4937ff/rate-test-model-4937ff-kserve-8459b77dfb-4rrg2"]
Sep 19 12:01:17.176500 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.176460    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml" podUID="39505ce3-63b9-4972-841f-42a92b4ad14d" containerName="main" containerID="cri-o://a7ee873b626ac9f40b078bc44588e5526df0c9c468e40e2ec6bb13b20195c5ce" gracePeriod=45
Sep 19 12:01:17.439308 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.439240    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:01:17.508237 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.508195    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-tmp-dir\") pod \"39505ce3-63b9-4972-841f-42a92b4ad14d\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") "
Sep 19 12:01:17.508425 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.508276    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-dshm\") pod \"39505ce3-63b9-4972-841f-42a92b4ad14d\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") "
Sep 19 12:01:17.508425 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.508309    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-home\") pod \"39505ce3-63b9-4972-841f-42a92b4ad14d\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") "
Sep 19 12:01:17.508425 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.508338    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39505ce3-63b9-4972-841f-42a92b4ad14d-tls-certs\") pod \"39505ce3-63b9-4972-841f-42a92b4ad14d\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") "
Sep 19 12:01:17.508425 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.508376    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5dxh\" (UniqueName: \"kubernetes.io/projected/39505ce3-63b9-4972-841f-42a92b4ad14d-kube-api-access-l5dxh\") pod \"39505ce3-63b9-4972-841f-42a92b4ad14d\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") "
Sep 19 12:01:17.508610 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.508440    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-model-cache\") pod \"39505ce3-63b9-4972-841f-42a92b4ad14d\" (UID: \"39505ce3-63b9-4972-841f-42a92b4ad14d\") "
Sep 19 12:01:17.508610 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.508567    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39505ce3-63b9-4972-841f-42a92b4ad14d" (UID: "39505ce3-63b9-4972-841f-42a92b4ad14d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:01:17.508698 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.508567    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-home" (OuterVolumeSpecName: "home") pod "39505ce3-63b9-4972-841f-42a92b4ad14d" (UID: "39505ce3-63b9-4972-841f-42a92b4ad14d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:01:17.508790 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.508769    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-model-cache" (OuterVolumeSpecName: "model-cache") pod "39505ce3-63b9-4972-841f-42a92b4ad14d" (UID: "39505ce3-63b9-4972-841f-42a92b4ad14d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:01:17.508847 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.508836    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-home\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:01:17.508888 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.508851    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-tmp-dir\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:01:17.510763 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.510728    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39505ce3-63b9-4972-841f-42a92b4ad14d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39505ce3-63b9-4972-841f-42a92b4ad14d" (UID: "39505ce3-63b9-4972-841f-42a92b4ad14d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 12:01:17.511196 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.511175    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-dshm" (OuterVolumeSpecName: "dshm") pod "39505ce3-63b9-4972-841f-42a92b4ad14d" (UID: "39505ce3-63b9-4972-841f-42a92b4ad14d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:01:17.511309 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.511194    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39505ce3-63b9-4972-841f-42a92b4ad14d-kube-api-access-l5dxh" (OuterVolumeSpecName: "kube-api-access-l5dxh") pod "39505ce3-63b9-4972-841f-42a92b4ad14d" (UID: "39505ce3-63b9-4972-841f-42a92b4ad14d"). InnerVolumeSpecName "kube-api-access-l5dxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 12:01:17.609399 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.609345    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-model-cache\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:01:17.609399 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.609391    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39505ce3-63b9-4972-841f-42a92b4ad14d-dshm\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:01:17.609399 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.609400    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39505ce3-63b9-4972-841f-42a92b4ad14d-tls-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:01:17.609399 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.609410    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5dxh\" (UniqueName: \"kubernetes.io/projected/39505ce3-63b9-4972-841f-42a92b4ad14d-kube-api-access-l5dxh\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:01:17.904871 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:17.904837    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d" path="/var/lib/kubelet/pods/ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d/volumes"
Sep 19 12:01:18.115709 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:18.115676    2584 generic.go:356] "Generic (PLEG): container finished" podID="39505ce3-63b9-4972-841f-42a92b4ad14d" containerID="a7ee873b626ac9f40b078bc44588e5526df0c9c468e40e2ec6bb13b20195c5ce" exitCode=0
Sep 19 12:01:18.115925 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:18.115717    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"
Sep 19 12:01:18.115925 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:18.115761    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml" event={"ID":"39505ce3-63b9-4972-841f-42a92b4ad14d","Type":"ContainerDied","Data":"a7ee873b626ac9f40b078bc44588e5526df0c9c468e40e2ec6bb13b20195c5ce"}
Sep 19 12:01:18.115925 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:18.115804    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml" event={"ID":"39505ce3-63b9-4972-841f-42a92b4ad14d","Type":"ContainerDied","Data":"082deedbf21c2f8a31a012bb78f6ca9b8e038b8453b3adbbe9a663b358fad222"}
Sep 19 12:01:18.115925 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:18.115820    2584 scope.go:117] "RemoveContainer" containerID="a7ee873b626ac9f40b078bc44588e5526df0c9c468e40e2ec6bb13b20195c5ce"
Sep 19 12:01:18.125426 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:18.125402    2584 scope.go:117] "RemoveContainer" containerID="a7ee873b626ac9f40b078bc44588e5526df0c9c468e40e2ec6bb13b20195c5ce"
Sep 19 12:01:18.125773 ip-10-0-133-209 kubenswrapper[2584]: E0919 12:01:18.125752    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7ee873b626ac9f40b078bc44588e5526df0c9c468e40e2ec6bb13b20195c5ce\": container with ID starting with a7ee873b626ac9f40b078bc44588e5526df0c9c468e40e2ec6bb13b20195c5ce not found: ID does not exist" containerID="a7ee873b626ac9f40b078bc44588e5526df0c9c468e40e2ec6bb13b20195c5ce"
Sep 19 12:01:18.125843 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:18.125782    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ee873b626ac9f40b078bc44588e5526df0c9c468e40e2ec6bb13b20195c5ce"} err="failed to get container status \"a7ee873b626ac9f40b078bc44588e5526df0c9c468e40e2ec6bb13b20195c5ce\": rpc error: code = NotFound desc = could not find container \"a7ee873b626ac9f40b078bc44588e5526df0c9c468e40e2ec6bb13b20195c5ce\": container with ID starting with a7ee873b626ac9f40b078bc44588e5526df0c9c468e40e2ec6bb13b20195c5ce not found: ID does not exist"
Sep 19 12:01:18.133893 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:18.133858    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"]
Sep 19 12:01:18.138338 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:18.138304    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a8a59/rate-test-model-4a8a59-kserve-664475b648-dd5ml"]
Sep 19 12:01:19.908057 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:19.908008    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39505ce3-63b9-4972-841f-42a92b4ad14d" path="/var/lib/kubelet/pods/39505ce3-63b9-4972-841f-42a92b4ad14d/volumes"
Sep 19 12:01:29.175415 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:29.175382    2584 generic.go:356] "Generic (PLEG): container finished" podID="82514835-6ded-4b36-96b1-6e260624de3f" containerID="c80e9979d2e67463f66c5eaa1866d33ab5719d9e6f16b7fdf7a2e14789268805" exitCode=0
Sep 19 12:01:29.175955 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:29.175457    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw" event={"ID":"82514835-6ded-4b36-96b1-6e260624de3f","Type":"ContainerDied","Data":"c80e9979d2e67463f66c5eaa1866d33ab5719d9e6f16b7fdf7a2e14789268805"}
Sep 19 12:01:29.175955 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:29.175902    2584 scope.go:117] "RemoveContainer" containerID="c80e9979d2e67463f66c5eaa1866d33ab5719d9e6f16b7fdf7a2e14789268805"
Sep 19 12:01:29.177029 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:29.177000    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 12:01:29.177101 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:29.177086    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 12:01:29.177138 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:29.177131    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 12:01:29.442177 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:29.442089    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:30.076451 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:30.076417    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:30.182723 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:30.182681    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw" event={"ID":"82514835-6ded-4b36-96b1-6e260624de3f","Type":"ContainerStarted","Data":"92f5dd31273ecf816a7ee2066c206f25d523b1f7c76d58bac1447bc7a31f8c4b"}
Sep 19 12:01:30.203930 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:30.203869    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:30.204422 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:30.204384    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 19 12:01:30.204562 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:30.204451    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw" podUID="82514835-6ded-4b36-96b1-6e260624de3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 19 12:01:30.877410 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:30.877376    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:30.983435 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:30.983403    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:31.205094 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:31.204981    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 19 12:01:31.205094 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:31.205057    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw" podUID="82514835-6ded-4b36-96b1-6e260624de3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 19 12:01:31.576365 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:31.576304    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:31.676887 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:31.676853    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:31.954330 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:31.954245    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"]
Sep 19 12:01:31.954630 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:31.954617    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39505ce3-63b9-4972-841f-42a92b4ad14d" containerName="main"
Sep 19 12:01:31.954673 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:31.954632    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="39505ce3-63b9-4972-841f-42a92b4ad14d" containerName="main"
Sep 19 12:01:31.954673 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:31.954657    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d" containerName="main"
Sep 19 12:01:31.954673 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:31.954662    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d" containerName="main"
Sep 19 12:01:31.954773 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:31.954717    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad8f4c73-cf9b-4168-907f-6ef8bfd7a23d" containerName="main"
Sep 19 12:01:31.954773 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:31.954728    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="39505ce3-63b9-4972-841f-42a92b4ad14d" containerName="main"
Sep 19 12:01:31.957880 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:31.957857    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:31.960471 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:31.960447    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-df64a0-openshift-default-dockercfg-6l2df\""
Sep 19 12:01:31.974033 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:31.974006    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"]
Sep 19 12:01:31.985847 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:31.985801    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:32.049559 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.049520    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-credential-socket\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.049745 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.049568    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-podinfo\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.049745 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.049656    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-workload-certs\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.049745 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.049699    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istiod-ca-cert\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.049745 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.049717    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vz4r\" (UniqueName: \"kubernetes.io/projected/7bd4a0cb-4a14-4382-895b-9af64aad5f72-kube-api-access-7vz4r\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.049887 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.049796    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-token\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.049887 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.049833    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-envoy\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.049887 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.049858    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-data\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.050022 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.049886    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-workload-socket\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.150688 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.150641    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istiod-ca-cert\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.150688 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.150687    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vz4r\" (UniqueName: \"kubernetes.io/projected/7bd4a0cb-4a14-4382-895b-9af64aad5f72-kube-api-access-7vz4r\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.151024 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.150722    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-token\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.151024 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.150740    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-envoy\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.151024 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.150763    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-data\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.151024 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.150797    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-workload-socket\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.151024 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.150826    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-credential-socket\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.151024 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.150854    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-podinfo\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.151024 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.150944    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-workload-certs\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.151378 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.151302    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-workload-socket\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.151378 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.151349    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-credential-socket\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.151469 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.151350    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-data\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.151615 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.151583    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-workload-certs\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.151759 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.151550    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istiod-ca-cert\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.153384 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.153365    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-envoy\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.153597 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.153578    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-podinfo\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.161957 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.161899    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:32.162258 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.162232    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-token\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.162494 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.162479    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vz4r\" (UniqueName: \"kubernetes.io/projected/7bd4a0cb-4a14-4382-895b-9af64aad5f72-kube-api-access-7vz4r\") pod \"e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") " pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.204068 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.204025    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 19 12:01:32.204273 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.204110    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw" podUID="82514835-6ded-4b36-96b1-6e260624de3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 19 12:01:32.269143 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.269032    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:32.572656 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.572629    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"]
Sep 19 12:01:32.574248 ip-10-0-133-209 kubenswrapper[2584]: W0919 12:01:32.574211    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd4a0cb_4a14_4382_895b_9af64aad5f72.slice/crio-1f850343f400753e10a0af87d583b919aa2b2d15050a3bb4faa0825ab562d46e WatchSource:0}: Error finding container 1f850343f400753e10a0af87d583b919aa2b2d15050a3bb4faa0825ab562d46e: Status 404 returned error can't find the container with id 1f850343f400753e10a0af87d583b919aa2b2d15050a3bb4faa0825ab562d46e
Sep 19 12:01:32.576524 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.576490    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 12:01:32.576606 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.576561    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 12:01:32.576606 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:32.576589    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 12:01:33.204744 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:33.204253    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 19 12:01:33.204744 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:33.204319    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw" podUID="82514835-6ded-4b36-96b1-6e260624de3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 19 12:01:33.212797 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:33.211325    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95" event={"ID":"7bd4a0cb-4a14-4382-895b-9af64aad5f72","Type":"ContainerStarted","Data":"3b608c4bd64804fc1d785d3aab2c0695f32d9b24ff7999a42acd04054155b891"}
Sep 19 12:01:33.212797 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:33.211393    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95" event={"ID":"7bd4a0cb-4a14-4382-895b-9af64aad5f72","Type":"ContainerStarted","Data":"1f850343f400753e10a0af87d583b919aa2b2d15050a3bb4faa0825ab562d46e"}
Sep 19 12:01:33.232590 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:33.232536    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95" podStartSLOduration=2.232516612 podStartE2EDuration="2.232516612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:01:33.231150155 +0000 UTC m=+1677.431179957" watchObservedRunningTime="2026-09-19 12:01:33.232516612 +0000 UTC m=+1677.432546415"
Sep 19 12:01:33.269698 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:33.269561    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:33.271074 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:33.271030    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 19 12:01:33.271217 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:33.271089    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95" podUID="7bd4a0cb-4a14-4382-895b-9af64aad5f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 19 12:01:34.204710 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:34.204671    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 19 12:01:34.205172 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:34.205082    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw" podUID="82514835-6ded-4b36-96b1-6e260624de3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 19 12:01:34.274083 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:34.274051    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:34.841705 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:34.841674    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:35.037592 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:35.037551    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:35.117572 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:35.117457    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:35.204504 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:35.204463    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 19 12:01:35.204818 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:35.204786    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw" podUID="82514835-6ded-4b36-96b1-6e260624de3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 19 12:01:35.218410 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:35.218385    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:35.223530 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:35.223512    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:35.224433 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:35.224415    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:35.600189 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:35.600134    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:36.208297 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:36.208266    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:36.209692 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:36.209667    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:01:36.263415 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:36.263385    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:37.175581 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:37.175546    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:37.236218 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:37.236187    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:37.399130 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:37.399093    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:37.628351 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:37.628320    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:37.725260 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:37.725224    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:51.265122 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:51.265073    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 19 12:01:51.265717 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:51.265189    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95" podUID="7bd4a0cb-4a14-4382-895b-9af64aad5f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 19 12:01:52.324458 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:52.324424    2584 generic.go:356] "Generic (PLEG): container finished" podID="7bd4a0cb-4a14-4382-895b-9af64aad5f72" containerID="3b608c4bd64804fc1d785d3aab2c0695f32d9b24ff7999a42acd04054155b891" exitCode=0
Sep 19 12:01:52.324887 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:52.324506    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95" event={"ID":"7bd4a0cb-4a14-4382-895b-9af64aad5f72","Type":"ContainerDied","Data":"3b608c4bd64804fc1d785d3aab2c0695f32d9b24ff7999a42acd04054155b891"}
Sep 19 12:01:52.324952 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:52.324926    2584 scope.go:117] "RemoveContainer" containerID="3b608c4bd64804fc1d785d3aab2c0695f32d9b24ff7999a42acd04054155b891"
Sep 19 12:01:52.326053 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:52.326018    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 12:01:52.326126 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:52.326084    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 12:01:52.326126 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:52.326117    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 12:01:53.333560 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:53.333514    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95" event={"ID":"7bd4a0cb-4a14-4382-895b-9af64aad5f72","Type":"ContainerStarted","Data":"ca05bdc1231d29cf7ca707632c18a156bf61b07f3b05fa31477b541c7f354551"}
Sep 19 12:01:54.269410 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:54.269369    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:01:54.269651 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:54.269630    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 19 12:01:54.269706 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:54.269680    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95" podUID="7bd4a0cb-4a14-4382-895b-9af64aad5f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 19 12:01:55.269550 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:55.269507    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 19 12:01:55.270011 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:55.269571    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95" podUID="7bd4a0cb-4a14-4382-895b-9af64aad5f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 19 12:01:56.270013 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:56.269975    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 19 12:01:56.270406 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:56.270034    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95" podUID="7bd4a0cb-4a14-4382-895b-9af64aad5f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 19 12:01:57.270244 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:57.270209    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 19 12:01:57.270638 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:57.270266    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95" podUID="7bd4a0cb-4a14-4382-895b-9af64aad5f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 19 12:01:58.270177 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:58.270138    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 19 12:01:58.270400 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:58.270202    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95" podUID="7bd4a0cb-4a14-4382-895b-9af64aad5f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 19 12:01:59.026045 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:59.026008    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:01:59.269810 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:59.269763    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 19 12:01:59.270056 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:59.269829    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95" podUID="7bd4a0cb-4a14-4382-895b-9af64aad5f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 19 12:01:59.753029 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:01:59.752988    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:00.173563 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:00.173484    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:00.273721 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:00.273684    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:02:00.273986 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:00.273965    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:02:00.274851 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:00.274834    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:02:00.384852 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:00.384814    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:01.049902 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:01.049852    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:02.040202 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:02.040165    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:02.655107 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:02.655043    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:08.837174 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:08.837137    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"]
Sep 19 12:02:08.840978 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:08.840957    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:08.843735 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:08.843701    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8a618a\"/\"openshift-service-ca.crt\""
Sep 19 12:02:08.845284 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:08.845160    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8a618a\"/\"kube-root-ca.crt\""
Sep 19 12:02:08.845882 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:08.845856    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8a618a\"/\"default-dockercfg-z9sl5\""
Sep 19 12:02:08.846026 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:08.845879    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8a618a\"/\"ipp-route-8a618a-kserve-self-signed-certs\""
Sep 19 12:02:08.849246 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:08.849215    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"]
Sep 19 12:02:08.859170 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:08.859139    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:08.924227 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:08.924186    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-tmp-dir\") pod \"ipp-route-8a618a-kserve-69db778fc6-bpb8t\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") " pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:08.924415 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:08.924245    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-home\") pod \"ipp-route-8a618a-kserve-69db778fc6-bpb8t\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") " pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:08.924415 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:08.924289    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-dshm\") pod \"ipp-route-8a618a-kserve-69db778fc6-bpb8t\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") " pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:08.924415 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:08.924336    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b444e909-bc58-42b7-9405-840b39e3584d-tls-certs\") pod \"ipp-route-8a618a-kserve-69db778fc6-bpb8t\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") " pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:08.924415 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:08.924393    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-model-cache\") pod \"ipp-route-8a618a-kserve-69db778fc6-bpb8t\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") " pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:08.924560 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:08.924443    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhfc5\" (UniqueName: \"kubernetes.io/projected/b444e909-bc58-42b7-9405-840b39e3584d-kube-api-access-bhfc5\") pod \"ipp-route-8a618a-kserve-69db778fc6-bpb8t\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") " pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:09.024972 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:09.024907    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-model-cache\") pod \"ipp-route-8a618a-kserve-69db778fc6-bpb8t\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") " pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:09.025176 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:09.025000    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhfc5\" (UniqueName: \"kubernetes.io/projected/b444e909-bc58-42b7-9405-840b39e3584d-kube-api-access-bhfc5\") pod \"ipp-route-8a618a-kserve-69db778fc6-bpb8t\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") " pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:09.025176 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:09.025051    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-tmp-dir\") pod \"ipp-route-8a618a-kserve-69db778fc6-bpb8t\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") " pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:09.025176 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:09.025099    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-home\") pod \"ipp-route-8a618a-kserve-69db778fc6-bpb8t\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") " pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:09.025176 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:09.025125    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-dshm\") pod \"ipp-route-8a618a-kserve-69db778fc6-bpb8t\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") " pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:09.025176 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:09.025146    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b444e909-bc58-42b7-9405-840b39e3584d-tls-certs\") pod \"ipp-route-8a618a-kserve-69db778fc6-bpb8t\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") " pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:09.025492 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:09.025339    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-model-cache\") pod \"ipp-route-8a618a-kserve-69db778fc6-bpb8t\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") " pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:09.025492 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:09.025461    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-home\") pod \"ipp-route-8a618a-kserve-69db778fc6-bpb8t\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") " pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:09.025492 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:09.025488    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-tmp-dir\") pod \"ipp-route-8a618a-kserve-69db778fc6-bpb8t\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") " pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:09.027767 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:09.027735    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-dshm\") pod \"ipp-route-8a618a-kserve-69db778fc6-bpb8t\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") " pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:09.027952 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:09.027935    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b444e909-bc58-42b7-9405-840b39e3584d-tls-certs\") pod \"ipp-route-8a618a-kserve-69db778fc6-bpb8t\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") " pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:09.033154 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:09.033128    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhfc5\" (UniqueName: \"kubernetes.io/projected/b444e909-bc58-42b7-9405-840b39e3584d-kube-api-access-bhfc5\") pod \"ipp-route-8a618a-kserve-69db778fc6-bpb8t\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") " pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:09.153312 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:09.153215    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:09.433059 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:09.433030    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"]
Sep 19 12:02:09.435129 ip-10-0-133-209 kubenswrapper[2584]: W0919 12:02:09.435095    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb444e909_bc58_42b7_9405_840b39e3584d.slice/crio-e64be7b4c39849b3dd7f11040a4c90d98f122ec9f2d3e07dd739fb8ef9ed0000 WatchSource:0}: Error finding container e64be7b4c39849b3dd7f11040a4c90d98f122ec9f2d3e07dd739fb8ef9ed0000: Status 404 returned error can't find the container with id e64be7b4c39849b3dd7f11040a4c90d98f122ec9f2d3e07dd739fb8ef9ed0000
Sep 19 12:02:10.430352 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:10.430311    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t" event={"ID":"b444e909-bc58-42b7-9405-840b39e3584d","Type":"ContainerStarted","Data":"b5fcf091826cc256697de48d54b529730d960512ca78db064e1b2e0ac92b276b"}
Sep 19 12:02:10.430784 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:10.430362    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t" event={"ID":"b444e909-bc58-42b7-9405-840b39e3584d","Type":"ContainerStarted","Data":"e64be7b4c39849b3dd7f11040a4c90d98f122ec9f2d3e07dd739fb8ef9ed0000"}
Sep 19 12:02:10.447768 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:10.447715    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t" podStartSLOduration=2.447698664 podStartE2EDuration="2.447698664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:02:10.445577647 +0000 UTC m=+1714.645607492" watchObservedRunningTime="2026-09-19 12:02:10.447698664 +0000 UTC m=+1714.647728467"
Sep 19 12:02:11.154033 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:11.153980    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:19.153603 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:19.153560    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:19.166455 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:19.166424    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:19.491249 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:19.491161    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:02:25.847826 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:25.847790    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:49.208871 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.208837    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:49.483247 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.483153    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"]
Sep 19 12:02:49.487010 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.486981    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.489678 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.489653    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-78dcba-openshift-default-dockercfg-xkp47\""
Sep 19 12:02:49.497032 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.497009    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"]
Sep 19 12:02:49.511272 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.511229    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:49.630907 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.630866    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-data\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.630907 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.630932    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg7b7\" (UniqueName: \"kubernetes.io/projected/6373b046-ef34-41d4-ac1e-afd01944cd12-kube-api-access-wg7b7\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.631162 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.631014    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-workload-socket\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.631162 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.631072    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-workload-certs\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.631162 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.631101    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-credential-socket\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.631162 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.631144    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-envoy\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.631298 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.631164    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-podinfo\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.631298 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.631199    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-token\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.631298 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.631238    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6373b046-ef34-41d4-ac1e-afd01944cd12-istiod-ca-cert\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.732356 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.732316    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-workload-socket\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.732557 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.732373    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-workload-certs\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.732557 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.732408    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-credential-socket\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.732557 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.732460    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-envoy\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.732557 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.732486    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-podinfo\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.732557 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.732530    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-token\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.732841 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.732561    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6373b046-ef34-41d4-ac1e-afd01944cd12-istiod-ca-cert\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.732841 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.732594    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-data\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.732841 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.732628    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg7b7\" (UniqueName: \"kubernetes.io/projected/6373b046-ef34-41d4-ac1e-afd01944cd12-kube-api-access-wg7b7\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.732841 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.732786    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-workload-socket\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.733110 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.732837    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-workload-certs\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.733110 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.733039    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-data\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.733206 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.733104    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-credential-socket\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.733470 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.733425    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6373b046-ef34-41d4-ac1e-afd01944cd12-istiod-ca-cert\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.735133 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.735113    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-envoy\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.735340 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.735323    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-podinfo\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.741629 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.741591    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-token\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.741783 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.741768    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg7b7\" (UniqueName: \"kubernetes.io/projected/6373b046-ef34-41d4-ac1e-afd01944cd12-kube-api-access-wg7b7\") pod \"e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") " pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:49.767689 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.767644    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:49.800219 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:49.800176    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:50.077993 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:50.077969    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"]
Sep 19 12:02:50.079686 ip-10-0-133-209 kubenswrapper[2584]: W0919 12:02:50.079659    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6373b046_ef34_41d4_ac1e_afd01944cd12.slice/crio-188ffc9c35f1dd99c667c8b5abea9d2eff9a197492c2258685be987a80d0fab5 WatchSource:0}: Error finding container 188ffc9c35f1dd99c667c8b5abea9d2eff9a197492c2258685be987a80d0fab5: Status 404 returned error can't find the container with id 188ffc9c35f1dd99c667c8b5abea9d2eff9a197492c2258685be987a80d0fab5
Sep 19 12:02:50.082456 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:50.082391    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 12:02:50.082536 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:50.082505    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 12:02:50.082575 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:50.082539    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 12:02:50.662275 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:50.662240    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5" event={"ID":"6373b046-ef34-41d4-ac1e-afd01944cd12","Type":"ContainerStarted","Data":"e33feb9b652e7c0a65082b8973be6beb5f422e70d53d0191052547ca3420ab68"}
Sep 19 12:02:50.662693 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:50.662301    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5" event={"ID":"6373b046-ef34-41d4-ac1e-afd01944cd12","Type":"ContainerStarted","Data":"188ffc9c35f1dd99c667c8b5abea9d2eff9a197492c2258685be987a80d0fab5"}
Sep 19 12:02:50.680960 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:50.680829    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5" podStartSLOduration=1.680807765 podStartE2EDuration="1.680807765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:02:50.680673468 +0000 UTC m=+1754.880703288" watchObservedRunningTime="2026-09-19 12:02:50.680807765 +0000 UTC m=+1754.880837586"
Sep 19 12:02:50.801084 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:50.801046    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:51.806881 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:51.806841    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:51.875681 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:51.875647    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:52.580963 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:52.580904    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:52.674525 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:52.674491    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:52.675474 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:52.675456    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:02:52.682651 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:52.682623    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:52.780948 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:52.780897    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:53.500746 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:53.500715    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:54.254475 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:54.254443    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:55.067435 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:55.067403    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:55.857227 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:55.857178    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:55.958535 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:55.958506    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:56.085486 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:56.085458    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:56.306579 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:56.306492    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:02:56.401653 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:02:56.401608    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:17.582551 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:17.582506    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:17.686189 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:17.686153    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:18.365365 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:18.365328    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:19.155677 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:19.155623    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:19.834633 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:19.834599    2584 generic.go:356] "Generic (PLEG): container finished" podID="6373b046-ef34-41d4-ac1e-afd01944cd12" containerID="e33feb9b652e7c0a65082b8973be6beb5f422e70d53d0191052547ca3420ab68" exitCode=0
Sep 19 12:03:19.834866 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:19.834652    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5" event={"ID":"6373b046-ef34-41d4-ac1e-afd01944cd12","Type":"ContainerDied","Data":"e33feb9b652e7c0a65082b8973be6beb5f422e70d53d0191052547ca3420ab68"}
Sep 19 12:03:19.835112 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:19.835097    2584 scope.go:117] "RemoveContainer" containerID="e33feb9b652e7c0a65082b8973be6beb5f422e70d53d0191052547ca3420ab68"
Sep 19 12:03:19.836431 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:19.836399    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 12:03:19.836511 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:19.836477    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 12:03:19.836553 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:19.836513    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 19 12:03:19.928016 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:19.927987    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:20.660369 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:20.660337    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:20.842260 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:20.842223    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5" event={"ID":"6373b046-ef34-41d4-ac1e-afd01944cd12","Type":"ContainerStarted","Data":"15a879851a0d9904b9030afbe090ed24923a7f8738571be88805302296787927"}
Sep 19 12:03:21.460367 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:21.460331    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:21.801784 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:21.801662    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:03:21.802420 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:21.802381    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 19 12:03:21.802552 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:21.802470    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5" podUID="6373b046-ef34-41d4-ac1e-afd01944cd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 19 12:03:22.246629 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:22.246596    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:22.675050 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:22.675010    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:03:22.801247 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:22.801199    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 19 12:03:22.801585 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:22.801550    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5" podUID="6373b046-ef34-41d4-ac1e-afd01944cd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 19 12:03:23.150247 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:23.150205    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:23.801280 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:23.801234    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 19 12:03:23.801513 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:23.801308    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5" podUID="6373b046-ef34-41d4-ac1e-afd01944cd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 19 12:03:23.966644 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:23.966614    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:24.528959 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:24.528829    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:24.628329 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:24.628302    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:24.721445 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:24.721414    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:24.781838 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:24.781751    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:24.801008 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:24.800968    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 19 12:03:24.801320 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:24.801283    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5" podUID="6373b046-ef34-41d4-ac1e-afd01944cd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 19 12:03:25.073199 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:25.073163    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:25.176712 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:25.176666    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:25.801031 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:25.800995    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 19 12:03:25.801664 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:25.801616    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5" podUID="6373b046-ef34-41d4-ac1e-afd01944cd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 19 12:03:25.850631 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:25.850591    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:26.554724 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:26.554687    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:26.805151 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:26.805068    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:03:26.806315 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:26.806289    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:03:27.370847 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:27.370815    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:27.999008 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:27.998971    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:30.949860 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:30.949826    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:31.007233 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:31.007194    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:31.249331 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:31.249237    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"]
Sep 19 12:03:31.249535 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:31.249489    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5" podUID="6373b046-ef34-41d4-ac1e-afd01944cd12" containerName="istio-proxy" containerID="cri-o://15a879851a0d9904b9030afbe090ed24923a7f8738571be88805302296787927" gracePeriod=30
Sep 19 12:03:31.262138 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:31.262113    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:35.145057 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:35.145022    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:35.866130 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:35.866099    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:36.527734 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.527659    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:03:36.658157 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.658122    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:36.697464 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.697432    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-podinfo\") pod \"6373b046-ef34-41d4-ac1e-afd01944cd12\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") "
Sep 19 12:03:36.697663 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.697492    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-credential-socket\") pod \"6373b046-ef34-41d4-ac1e-afd01944cd12\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") "
Sep 19 12:03:36.697663 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.697516    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-workload-certs\") pod \"6373b046-ef34-41d4-ac1e-afd01944cd12\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") "
Sep 19 12:03:36.697663 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.697538    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-envoy\") pod \"6373b046-ef34-41d4-ac1e-afd01944cd12\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") "
Sep 19 12:03:36.697663 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.697593    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-token\") pod \"6373b046-ef34-41d4-ac1e-afd01944cd12\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") "
Sep 19 12:03:36.697663 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.697620    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-data\") pod \"6373b046-ef34-41d4-ac1e-afd01944cd12\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") "
Sep 19 12:03:36.697663 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.697655    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wg7b7\" (UniqueName: \"kubernetes.io/projected/6373b046-ef34-41d4-ac1e-afd01944cd12-kube-api-access-wg7b7\") pod \"6373b046-ef34-41d4-ac1e-afd01944cd12\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") "
Sep 19 12:03:36.698009 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.697682    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-workload-socket\") pod \"6373b046-ef34-41d4-ac1e-afd01944cd12\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") "
Sep 19 12:03:36.698009 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.697760    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6373b046-ef34-41d4-ac1e-afd01944cd12-istiod-ca-cert\") pod \"6373b046-ef34-41d4-ac1e-afd01944cd12\" (UID: \"6373b046-ef34-41d4-ac1e-afd01944cd12\") "
Sep 19 12:03:36.698009 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.697814    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6373b046-ef34-41d4-ac1e-afd01944cd12" (UID: "6373b046-ef34-41d4-ac1e-afd01944cd12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:03:36.698009 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.697864    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6373b046-ef34-41d4-ac1e-afd01944cd12" (UID: "6373b046-ef34-41d4-ac1e-afd01944cd12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:03:36.698209 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.698050    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6373b046-ef34-41d4-ac1e-afd01944cd12" (UID: "6373b046-ef34-41d4-ac1e-afd01944cd12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:03:36.698209 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.698071    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-data" (OuterVolumeSpecName: "istio-data") pod "6373b046-ef34-41d4-ac1e-afd01944cd12" (UID: "6373b046-ef34-41d4-ac1e-afd01944cd12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:03:36.698319 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.698203    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6373b046-ef34-41d4-ac1e-afd01944cd12-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6373b046-ef34-41d4-ac1e-afd01944cd12" (UID: "6373b046-ef34-41d4-ac1e-afd01944cd12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 12:03:36.698319 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.698208    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-credential-socket\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:03:36.698319 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.698240    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-workload-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:03:36.698319 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.698254    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-data\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:03:36.698319 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.698266    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-workload-socket\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:03:36.700224 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.700189    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6373b046-ef34-41d4-ac1e-afd01944cd12" (UID: "6373b046-ef34-41d4-ac1e-afd01944cd12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:03:36.700353 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.700271    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6373b046-ef34-41d4-ac1e-afd01944cd12-kube-api-access-wg7b7" (OuterVolumeSpecName: "kube-api-access-wg7b7") pod "6373b046-ef34-41d4-ac1e-afd01944cd12" (UID: "6373b046-ef34-41d4-ac1e-afd01944cd12"). InnerVolumeSpecName "kube-api-access-wg7b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 12:03:36.700353 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.700297    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6373b046-ef34-41d4-ac1e-afd01944cd12" (UID: "6373b046-ef34-41d4-ac1e-afd01944cd12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 12:03:36.700443 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.700368    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-token" (OuterVolumeSpecName: "istio-token") pod "6373b046-ef34-41d4-ac1e-afd01944cd12" (UID: "6373b046-ef34-41d4-ac1e-afd01944cd12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 12:03:36.799379 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.799275    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-envoy\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:03:36.799379 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.799310    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-token\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:03:36.799379 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.799324    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg7b7\" (UniqueName: \"kubernetes.io/projected/6373b046-ef34-41d4-ac1e-afd01944cd12-kube-api-access-wg7b7\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:03:36.799379 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.799336    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6373b046-ef34-41d4-ac1e-afd01944cd12-istiod-ca-cert\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:03:36.799379 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.799348    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6373b046-ef34-41d4-ac1e-afd01944cd12-istio-podinfo\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:03:36.935439 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.935397    2584 generic.go:356] "Generic (PLEG): container finished" podID="6373b046-ef34-41d4-ac1e-afd01944cd12" containerID="15a879851a0d9904b9030afbe090ed24923a7f8738571be88805302296787927" exitCode=0
Sep 19 12:03:36.935644 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.935481    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5" event={"ID":"6373b046-ef34-41d4-ac1e-afd01944cd12","Type":"ContainerDied","Data":"15a879851a0d9904b9030afbe090ed24923a7f8738571be88805302296787927"}
Sep 19 12:03:36.935644 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.935500    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"
Sep 19 12:03:36.935644 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.935539    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5" event={"ID":"6373b046-ef34-41d4-ac1e-afd01944cd12","Type":"ContainerDied","Data":"188ffc9c35f1dd99c667c8b5abea9d2eff9a197492c2258685be987a80d0fab5"}
Sep 19 12:03:36.935644 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.935569    2584 scope.go:117] "RemoveContainer" containerID="15a879851a0d9904b9030afbe090ed24923a7f8738571be88805302296787927"
Sep 19 12:03:36.946134 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.946109    2584 scope.go:117] "RemoveContainer" containerID="e33feb9b652e7c0a65082b8973be6beb5f422e70d53d0191052547ca3420ab68"
Sep 19 12:03:36.956210 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.956189    2584 scope.go:117] "RemoveContainer" containerID="15a879851a0d9904b9030afbe090ed24923a7f8738571be88805302296787927"
Sep 19 12:03:36.956548 ip-10-0-133-209 kubenswrapper[2584]: E0919 12:03:36.956524    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15a879851a0d9904b9030afbe090ed24923a7f8738571be88805302296787927\": container with ID starting with 15a879851a0d9904b9030afbe090ed24923a7f8738571be88805302296787927 not found: ID does not exist" containerID="15a879851a0d9904b9030afbe090ed24923a7f8738571be88805302296787927"
Sep 19 12:03:36.956622 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.956553    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a879851a0d9904b9030afbe090ed24923a7f8738571be88805302296787927"} err="failed to get container status \"15a879851a0d9904b9030afbe090ed24923a7f8738571be88805302296787927\": rpc error: code = NotFound desc = could not find container \"15a879851a0d9904b9030afbe090ed24923a7f8738571be88805302296787927\": container with ID starting with 15a879851a0d9904b9030afbe090ed24923a7f8738571be88805302296787927 not found: ID does not exist"
Sep 19 12:03:36.956622 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.956571    2584 scope.go:117] "RemoveContainer" containerID="e33feb9b652e7c0a65082b8973be6beb5f422e70d53d0191052547ca3420ab68"
Sep 19 12:03:36.956863 ip-10-0-133-209 kubenswrapper[2584]: E0919 12:03:36.956846    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e33feb9b652e7c0a65082b8973be6beb5f422e70d53d0191052547ca3420ab68\": container with ID starting with e33feb9b652e7c0a65082b8973be6beb5f422e70d53d0191052547ca3420ab68 not found: ID does not exist" containerID="e33feb9b652e7c0a65082b8973be6beb5f422e70d53d0191052547ca3420ab68"
Sep 19 12:03:36.956935 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.956867    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e33feb9b652e7c0a65082b8973be6beb5f422e70d53d0191052547ca3420ab68"} err="failed to get container status \"e33feb9b652e7c0a65082b8973be6beb5f422e70d53d0191052547ca3420ab68\": rpc error: code = NotFound desc = could not find container \"e33feb9b652e7c0a65082b8973be6beb5f422e70d53d0191052547ca3420ab68\": container with ID starting with e33feb9b652e7c0a65082b8973be6beb5f422e70d53d0191052547ca3420ab68 not found: ID does not exist"
Sep 19 12:03:36.958061 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.958041    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"]
Sep 19 12:03:36.962514 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:36.962488    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-78dcba-openshift-default-7c44747958-tbcm5"]
Sep 19 12:03:37.469088 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:37.469055    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:37.905317 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:37.905284    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6373b046-ef34-41d4-ac1e-afd01944cd12" path="/var/lib/kubelet/pods/6373b046-ef34-41d4-ac1e-afd01944cd12/volumes"
Sep 19 12:03:38.002062 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:38.002037    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/1.log"
Sep 19 12:03:38.005813 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:38.005789    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/1.log"
Sep 19 12:03:38.008802 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:38.008774    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6th98_2fed27e0-833f-49a8-b11e-8f61df6e6bd5/ovn-acl-logging/0.log"
Sep 19 12:03:38.012207 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:38.012190    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6th98_2fed27e0-833f-49a8-b11e-8f61df6e6bd5/ovn-acl-logging/0.log"
Sep 19 12:03:38.220707 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:38.220675    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:38.959555 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:38.959521    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:50.256762 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.256719    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"]
Sep 19 12:03:50.257378 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.256992    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg" podUID="49132f35-53b0-42fb-977c-f0566ce60304" containerName="maas-api" containerID="cri-o://b44d7611f959c9777453b1aea8096b029dbe6ab8eb16559e829eaca6c089493e" gracePeriod=30
Sep 19 12:03:50.289802 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.289770    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:50.349592 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.349551    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:50.406496 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.406456    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:50.465643 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.465577    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:50.515714 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.515673    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:50.523771 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.523743    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"
Sep 19 12:03:50.626200 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.626161    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49132f35-53b0-42fb-977c-f0566ce60304-maas-api-tls\") pod \"49132f35-53b0-42fb-977c-f0566ce60304\" (UID: \"49132f35-53b0-42fb-977c-f0566ce60304\") "
Sep 19 12:03:50.626394 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.626219    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49132f35-53b0-42fb-977c-f0566ce60304-openshift-service-ca\") pod \"49132f35-53b0-42fb-977c-f0566ce60304\" (UID: \"49132f35-53b0-42fb-977c-f0566ce60304\") "
Sep 19 12:03:50.626394 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.626272    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49132f35-53b0-42fb-977c-f0566ce60304-metrics-certs\") pod \"49132f35-53b0-42fb-977c-f0566ce60304\" (UID: \"49132f35-53b0-42fb-977c-f0566ce60304\") "
Sep 19 12:03:50.626394 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.626326    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ncgz\" (UniqueName: \"kubernetes.io/projected/49132f35-53b0-42fb-977c-f0566ce60304-kube-api-access-6ncgz\") pod \"49132f35-53b0-42fb-977c-f0566ce60304\" (UID: \"49132f35-53b0-42fb-977c-f0566ce60304\") "
Sep 19 12:03:50.626646 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.626613    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49132f35-53b0-42fb-977c-f0566ce60304-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "49132f35-53b0-42fb-977c-f0566ce60304" (UID: "49132f35-53b0-42fb-977c-f0566ce60304"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 12:03:50.628573 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.628545    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49132f35-53b0-42fb-977c-f0566ce60304-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "49132f35-53b0-42fb-977c-f0566ce60304" (UID: "49132f35-53b0-42fb-977c-f0566ce60304"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 12:03:50.628692 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.628599    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49132f35-53b0-42fb-977c-f0566ce60304-kube-api-access-6ncgz" (OuterVolumeSpecName: "kube-api-access-6ncgz") pod "49132f35-53b0-42fb-977c-f0566ce60304" (UID: "49132f35-53b0-42fb-977c-f0566ce60304"). InnerVolumeSpecName "kube-api-access-6ncgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 12:03:50.628692 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.628631    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49132f35-53b0-42fb-977c-f0566ce60304-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "49132f35-53b0-42fb-977c-f0566ce60304" (UID: "49132f35-53b0-42fb-977c-f0566ce60304"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 12:03:50.728119 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.728088    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49132f35-53b0-42fb-977c-f0566ce60304-maas-api-tls\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:03:50.728119 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.728117    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49132f35-53b0-42fb-977c-f0566ce60304-openshift-service-ca\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:03:50.728119 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.728126    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49132f35-53b0-42fb-977c-f0566ce60304-metrics-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:03:50.728363 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.728134    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ncgz\" (UniqueName: \"kubernetes.io/projected/49132f35-53b0-42fb-977c-f0566ce60304-kube-api-access-6ncgz\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:03:50.845461 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.845407    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:50.949930 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:50.949880    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:51.013890 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:51.013852    2584 generic.go:356] "Generic (PLEG): container finished" podID="49132f35-53b0-42fb-977c-f0566ce60304" containerID="b44d7611f959c9777453b1aea8096b029dbe6ab8eb16559e829eaca6c089493e" exitCode=0
Sep 19 12:03:51.014131 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:51.013948    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"
Sep 19 12:03:51.014131 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:51.013960    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg" event={"ID":"49132f35-53b0-42fb-977c-f0566ce60304","Type":"ContainerDied","Data":"b44d7611f959c9777453b1aea8096b029dbe6ab8eb16559e829eaca6c089493e"}
Sep 19 12:03:51.014131 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:51.014002    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg" event={"ID":"49132f35-53b0-42fb-977c-f0566ce60304","Type":"ContainerDied","Data":"dc12b9592e5ad2001f4dac351e7e94b94e659a0347a68ae1992cb84b33e1097e"}
Sep 19 12:03:51.014131 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:51.014035    2584 scope.go:117] "RemoveContainer" containerID="b44d7611f959c9777453b1aea8096b029dbe6ab8eb16559e829eaca6c089493e"
Sep 19 12:03:51.027557 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:51.027535    2584 scope.go:117] "RemoveContainer" containerID="b44d7611f959c9777453b1aea8096b029dbe6ab8eb16559e829eaca6c089493e"
Sep 19 12:03:51.027978 ip-10-0-133-209 kubenswrapper[2584]: E0919 12:03:51.027956    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b44d7611f959c9777453b1aea8096b029dbe6ab8eb16559e829eaca6c089493e\": container with ID starting with b44d7611f959c9777453b1aea8096b029dbe6ab8eb16559e829eaca6c089493e not found: ID does not exist" containerID="b44d7611f959c9777453b1aea8096b029dbe6ab8eb16559e829eaca6c089493e"
Sep 19 12:03:51.028046 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:51.027987    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b44d7611f959c9777453b1aea8096b029dbe6ab8eb16559e829eaca6c089493e"} err="failed to get container status \"b44d7611f959c9777453b1aea8096b029dbe6ab8eb16559e829eaca6c089493e\": rpc error: code = NotFound desc = could not find container \"b44d7611f959c9777453b1aea8096b029dbe6ab8eb16559e829eaca6c089493e\": container with ID starting with b44d7611f959c9777453b1aea8096b029dbe6ab8eb16559e829eaca6c089493e not found: ID does not exist"
Sep 19 12:03:51.039277 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:51.039243    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"]
Sep 19 12:03:51.043261 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:51.043223    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8a618a-7759b877bd-v8mxg"]
Sep 19 12:03:51.768471 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:51.768425    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:51.904185 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:51.904148    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49132f35-53b0-42fb-977c-f0566ce60304" path="/var/lib/kubelet/pods/49132f35-53b0-42fb-977c-f0566ce60304/volumes"
Sep 19 12:03:52.531507 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:52.531470    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:53.185772 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:53.185727    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:03:55.826649 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:55.826603    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"]
Sep 19 12:03:55.865528 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:03:55.865495    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:10.847542 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:10.847462    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t" podUID="b444e909-bc58-42b7-9405-840b39e3584d" containerName="main" containerID="cri-o://b5fcf091826cc256697de48d54b529730d960512ca78db064e1b2e0ac92b276b" gracePeriod=45
Sep 19 12:04:11.094778 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.094746    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:04:11.131294 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.131183    2584 generic.go:356] "Generic (PLEG): container finished" podID="b444e909-bc58-42b7-9405-840b39e3584d" containerID="b5fcf091826cc256697de48d54b529730d960512ca78db064e1b2e0ac92b276b" exitCode=0
Sep 19 12:04:11.131294 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.131221    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"
Sep 19 12:04:11.131527 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.131288    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t" event={"ID":"b444e909-bc58-42b7-9405-840b39e3584d","Type":"ContainerDied","Data":"b5fcf091826cc256697de48d54b529730d960512ca78db064e1b2e0ac92b276b"}
Sep 19 12:04:11.131527 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.131352    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t" event={"ID":"b444e909-bc58-42b7-9405-840b39e3584d","Type":"ContainerDied","Data":"e64be7b4c39849b3dd7f11040a4c90d98f122ec9f2d3e07dd739fb8ef9ed0000"}
Sep 19 12:04:11.131527 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.131378    2584 scope.go:117] "RemoveContainer" containerID="b5fcf091826cc256697de48d54b529730d960512ca78db064e1b2e0ac92b276b"
Sep 19 12:04:11.141098 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.141077    2584 scope.go:117] "RemoveContainer" containerID="b5fcf091826cc256697de48d54b529730d960512ca78db064e1b2e0ac92b276b"
Sep 19 12:04:11.141442 ip-10-0-133-209 kubenswrapper[2584]: E0919 12:04:11.141421    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5fcf091826cc256697de48d54b529730d960512ca78db064e1b2e0ac92b276b\": container with ID starting with b5fcf091826cc256697de48d54b529730d960512ca78db064e1b2e0ac92b276b not found: ID does not exist" containerID="b5fcf091826cc256697de48d54b529730d960512ca78db064e1b2e0ac92b276b"
Sep 19 12:04:11.141525 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.141455    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5fcf091826cc256697de48d54b529730d960512ca78db064e1b2e0ac92b276b"} err="failed to get container status \"b5fcf091826cc256697de48d54b529730d960512ca78db064e1b2e0ac92b276b\": rpc error: code = NotFound desc = could not find container \"b5fcf091826cc256697de48d54b529730d960512ca78db064e1b2e0ac92b276b\": container with ID starting with b5fcf091826cc256697de48d54b529730d960512ca78db064e1b2e0ac92b276b not found: ID does not exist"
Sep 19 12:04:11.233269 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.233232    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-dshm\") pod \"b444e909-bc58-42b7-9405-840b39e3584d\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") "
Sep 19 12:04:11.233475 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.233306    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-home\") pod \"b444e909-bc58-42b7-9405-840b39e3584d\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") "
Sep 19 12:04:11.233475 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.233347    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-model-cache\") pod \"b444e909-bc58-42b7-9405-840b39e3584d\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") "
Sep 19 12:04:11.233475 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.233368    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b444e909-bc58-42b7-9405-840b39e3584d-tls-certs\") pod \"b444e909-bc58-42b7-9405-840b39e3584d\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") "
Sep 19 12:04:11.233475 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.233428    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-tmp-dir\") pod \"b444e909-bc58-42b7-9405-840b39e3584d\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") "
Sep 19 12:04:11.233475 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.233460    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhfc5\" (UniqueName: \"kubernetes.io/projected/b444e909-bc58-42b7-9405-840b39e3584d-kube-api-access-bhfc5\") pod \"b444e909-bc58-42b7-9405-840b39e3584d\" (UID: \"b444e909-bc58-42b7-9405-840b39e3584d\") "
Sep 19 12:04:11.233761 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.233614    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-home" (OuterVolumeSpecName: "home") pod "b444e909-bc58-42b7-9405-840b39e3584d" (UID: "b444e909-bc58-42b7-9405-840b39e3584d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:04:11.233761 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.233668    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-model-cache" (OuterVolumeSpecName: "model-cache") pod "b444e909-bc58-42b7-9405-840b39e3584d" (UID: "b444e909-bc58-42b7-9405-840b39e3584d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:04:11.233761 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.233745    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-home\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:04:11.233761 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.233751    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b444e909-bc58-42b7-9405-840b39e3584d" (UID: "b444e909-bc58-42b7-9405-840b39e3584d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:04:11.235846 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.235817    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b444e909-bc58-42b7-9405-840b39e3584d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b444e909-bc58-42b7-9405-840b39e3584d" (UID: "b444e909-bc58-42b7-9405-840b39e3584d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 12:04:11.235994 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.235850    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-dshm" (OuterVolumeSpecName: "dshm") pod "b444e909-bc58-42b7-9405-840b39e3584d" (UID: "b444e909-bc58-42b7-9405-840b39e3584d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:04:11.235994 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.235864    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b444e909-bc58-42b7-9405-840b39e3584d-kube-api-access-bhfc5" (OuterVolumeSpecName: "kube-api-access-bhfc5") pod "b444e909-bc58-42b7-9405-840b39e3584d" (UID: "b444e909-bc58-42b7-9405-840b39e3584d"). InnerVolumeSpecName "kube-api-access-bhfc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 12:04:11.334384 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.334347    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-tmp-dir\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:04:11.334384 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.334376    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhfc5\" (UniqueName: \"kubernetes.io/projected/b444e909-bc58-42b7-9405-840b39e3584d-kube-api-access-bhfc5\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:04:11.334384 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.334385    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-dshm\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:04:11.334384 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.334394    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b444e909-bc58-42b7-9405-840b39e3584d-model-cache\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:04:11.334766 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.334401    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b444e909-bc58-42b7-9405-840b39e3584d-tls-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:04:11.453548 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.453508    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"]
Sep 19 12:04:11.461714 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.457243    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-8a618a/ipp-route-8a618a-kserve-69db778fc6-bpb8t"]
Sep 19 12:04:11.903785 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:11.903745    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b444e909-bc58-42b7-9405-840b39e3584d" path="/var/lib/kubelet/pods/b444e909-bc58-42b7-9405-840b39e3584d/volumes"
Sep 19 12:04:32.504088 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:32.504040    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:33.688301 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:33.688261    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:38.290504 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:38.290463    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:38.391436 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:38.391406    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:38.550620 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:38.550508    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"]
Sep 19 12:04:38.550875 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:38.550846    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw" podUID="82514835-6ded-4b36-96b1-6e260624de3f" containerName="istio-proxy" containerID="cri-o://92f5dd31273ecf816a7ee2066c206f25d523b1f7c76d58bac1447bc7a31f8c4b" gracePeriod=30
Sep 19 12:04:38.568356 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:38.568314    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:39.651627 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:39.651590    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:40.558748 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:40.558709    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:41.232242 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:41.232207    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:41.966332 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:41.966300    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:43.815861 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.815785    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:04:43.949192 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.949158    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-istio-data\") pod \"82514835-6ded-4b36-96b1-6e260624de3f\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") "
Sep 19 12:04:43.949393 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.949199    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82514835-6ded-4b36-96b1-6e260624de3f-istio-token\") pod \"82514835-6ded-4b36-96b1-6e260624de3f\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") "
Sep 19 12:04:43.949393 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.949242    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-credential-socket\") pod \"82514835-6ded-4b36-96b1-6e260624de3f\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") "
Sep 19 12:04:43.949393 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.949294    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82514835-6ded-4b36-96b1-6e260624de3f-istiod-ca-cert\") pod \"82514835-6ded-4b36-96b1-6e260624de3f\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") "
Sep 19 12:04:43.949393 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.949343    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgfqd\" (UniqueName: \"kubernetes.io/projected/82514835-6ded-4b36-96b1-6e260624de3f-kube-api-access-kgfqd\") pod \"82514835-6ded-4b36-96b1-6e260624de3f\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") "
Sep 19 12:04:43.949393 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.949365    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-istio-envoy\") pod \"82514835-6ded-4b36-96b1-6e260624de3f\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") "
Sep 19 12:04:43.949673 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.949420    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-workload-socket\") pod \"82514835-6ded-4b36-96b1-6e260624de3f\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") "
Sep 19 12:04:43.949673 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.949457    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-istio-data" (OuterVolumeSpecName: "istio-data") pod "82514835-6ded-4b36-96b1-6e260624de3f" (UID: "82514835-6ded-4b36-96b1-6e260624de3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:04:43.949673 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.949471    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-workload-certs\") pod \"82514835-6ded-4b36-96b1-6e260624de3f\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") "
Sep 19 12:04:43.949673 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.949537    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82514835-6ded-4b36-96b1-6e260624de3f-istio-podinfo\") pod \"82514835-6ded-4b36-96b1-6e260624de3f\" (UID: \"82514835-6ded-4b36-96b1-6e260624de3f\") "
Sep 19 12:04:43.949861 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.949534    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "82514835-6ded-4b36-96b1-6e260624de3f" (UID: "82514835-6ded-4b36-96b1-6e260624de3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:04:43.949861 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.949704    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "82514835-6ded-4b36-96b1-6e260624de3f" (UID: "82514835-6ded-4b36-96b1-6e260624de3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:04:43.949861 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.949803    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82514835-6ded-4b36-96b1-6e260624de3f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "82514835-6ded-4b36-96b1-6e260624de3f" (UID: "82514835-6ded-4b36-96b1-6e260624de3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 12:04:43.949861 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.949738    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "82514835-6ded-4b36-96b1-6e260624de3f" (UID: "82514835-6ded-4b36-96b1-6e260624de3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:04:43.950101 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.949937    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-workload-socket\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:04:43.950101 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.949959    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-workload-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:04:43.950101 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.949973    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-istio-data\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:04:43.950101 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.949986    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-credential-socket\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:04:43.950101 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.949997    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82514835-6ded-4b36-96b1-6e260624de3f-istiod-ca-cert\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:04:43.951811 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.951787    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82514835-6ded-4b36-96b1-6e260624de3f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "82514835-6ded-4b36-96b1-6e260624de3f" (UID: "82514835-6ded-4b36-96b1-6e260624de3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 12:04:43.951960 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.951817    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "82514835-6ded-4b36-96b1-6e260624de3f" (UID: "82514835-6ded-4b36-96b1-6e260624de3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:04:43.951960 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.951845    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82514835-6ded-4b36-96b1-6e260624de3f-istio-token" (OuterVolumeSpecName: "istio-token") pod "82514835-6ded-4b36-96b1-6e260624de3f" (UID: "82514835-6ded-4b36-96b1-6e260624de3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 12:04:43.951960 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:43.951951    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82514835-6ded-4b36-96b1-6e260624de3f-kube-api-access-kgfqd" (OuterVolumeSpecName: "kube-api-access-kgfqd") pod "82514835-6ded-4b36-96b1-6e260624de3f" (UID: "82514835-6ded-4b36-96b1-6e260624de3f"). InnerVolumeSpecName "kube-api-access-kgfqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 12:04:44.050467 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:44.050430    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82514835-6ded-4b36-96b1-6e260624de3f-istio-podinfo\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:04:44.050467 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:44.050458    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82514835-6ded-4b36-96b1-6e260624de3f-istio-token\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:04:44.050467 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:44.050470    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgfqd\" (UniqueName: \"kubernetes.io/projected/82514835-6ded-4b36-96b1-6e260624de3f-kube-api-access-kgfqd\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:04:44.050467 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:44.050479    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82514835-6ded-4b36-96b1-6e260624de3f-istio-envoy\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:04:44.307543 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:44.307452    2584 generic.go:356] "Generic (PLEG): container finished" podID="82514835-6ded-4b36-96b1-6e260624de3f" containerID="92f5dd31273ecf816a7ee2066c206f25d523b1f7c76d58bac1447bc7a31f8c4b" exitCode=0
Sep 19 12:04:44.307543 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:44.307520    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"
Sep 19 12:04:44.307733 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:44.307540    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw" event={"ID":"82514835-6ded-4b36-96b1-6e260624de3f","Type":"ContainerDied","Data":"92f5dd31273ecf816a7ee2066c206f25d523b1f7c76d58bac1447bc7a31f8c4b"}
Sep 19 12:04:44.307733 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:44.307622    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw" event={"ID":"82514835-6ded-4b36-96b1-6e260624de3f","Type":"ContainerDied","Data":"dbdb19ce7dbe4fbc659cfaf2ccb339b907f4314070d849d72dadef676a1cf5b3"}
Sep 19 12:04:44.307733 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:44.307644    2584 scope.go:117] "RemoveContainer" containerID="92f5dd31273ecf816a7ee2066c206f25d523b1f7c76d58bac1447bc7a31f8c4b"
Sep 19 12:04:44.317703 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:44.317685    2584 scope.go:117] "RemoveContainer" containerID="c80e9979d2e67463f66c5eaa1866d33ab5719d9e6f16b7fdf7a2e14789268805"
Sep 19 12:04:44.326692 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:44.326665    2584 scope.go:117] "RemoveContainer" containerID="92f5dd31273ecf816a7ee2066c206f25d523b1f7c76d58bac1447bc7a31f8c4b"
Sep 19 12:04:44.327374 ip-10-0-133-209 kubenswrapper[2584]: E0919 12:04:44.327344    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f5dd31273ecf816a7ee2066c206f25d523b1f7c76d58bac1447bc7a31f8c4b\": container with ID starting with 92f5dd31273ecf816a7ee2066c206f25d523b1f7c76d58bac1447bc7a31f8c4b not found: ID does not exist" containerID="92f5dd31273ecf816a7ee2066c206f25d523b1f7c76d58bac1447bc7a31f8c4b"
Sep 19 12:04:44.327492 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:44.327386    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f5dd31273ecf816a7ee2066c206f25d523b1f7c76d58bac1447bc7a31f8c4b"} err="failed to get container status \"92f5dd31273ecf816a7ee2066c206f25d523b1f7c76d58bac1447bc7a31f8c4b\": rpc error: code = NotFound desc = could not find container \"92f5dd31273ecf816a7ee2066c206f25d523b1f7c76d58bac1447bc7a31f8c4b\": container with ID starting with 92f5dd31273ecf816a7ee2066c206f25d523b1f7c76d58bac1447bc7a31f8c4b not found: ID does not exist"
Sep 19 12:04:44.327492 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:44.327411    2584 scope.go:117] "RemoveContainer" containerID="c80e9979d2e67463f66c5eaa1866d33ab5719d9e6f16b7fdf7a2e14789268805"
Sep 19 12:04:44.327749 ip-10-0-133-209 kubenswrapper[2584]: E0919 12:04:44.327726    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80e9979d2e67463f66c5eaa1866d33ab5719d9e6f16b7fdf7a2e14789268805\": container with ID starting with c80e9979d2e67463f66c5eaa1866d33ab5719d9e6f16b7fdf7a2e14789268805 not found: ID does not exist" containerID="c80e9979d2e67463f66c5eaa1866d33ab5719d9e6f16b7fdf7a2e14789268805"
Sep 19 12:04:44.327796 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:44.327760    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80e9979d2e67463f66c5eaa1866d33ab5719d9e6f16b7fdf7a2e14789268805"} err="failed to get container status \"c80e9979d2e67463f66c5eaa1866d33ab5719d9e6f16b7fdf7a2e14789268805\": rpc error: code = NotFound desc = could not find container \"c80e9979d2e67463f66c5eaa1866d33ab5719d9e6f16b7fdf7a2e14789268805\": container with ID starting with c80e9979d2e67463f66c5eaa1866d33ab5719d9e6f16b7fdf7a2e14789268805 not found: ID does not exist"
Sep 19 12:04:44.328962 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:44.328940    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"]
Sep 19 12:04:44.333084 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:44.333050    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-8a618a-openshift-default-7f7b7d4db4-bn4rw"]
Sep 19 12:04:44.914269 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:44.914230    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:45.903380 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:45.903344    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82514835-6ded-4b36-96b1-6e260624de3f" path="/var/lib/kubelet/pods/82514835-6ded-4b36-96b1-6e260624de3f/volumes"
Sep 19 12:04:49.369368 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:49.369295    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:49.440653 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:49.440623    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:49.495457 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:49.495419    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:49.565338 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:49.565305    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:49.622218 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:49.622093    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:49.948902 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:49.948775    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:50.058332 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:50.058288    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:50.756466 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:50.756428    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:51.529718 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:51.529686    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:55.856400 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:55.856367    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:55.913618 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:55.913587    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:56.103329 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:56.103285    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"]
Sep 19 12:04:56.103621 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:56.103597    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95" podUID="7bd4a0cb-4a14-4382-895b-9af64aad5f72" containerName="istio-proxy" containerID="cri-o://ca05bdc1231d29cf7ca707632c18a156bf61b07f3b05fa31477b541c7f354551" gracePeriod=30
Sep 19 12:04:56.126144 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:56.126059    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:56.627787 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:56.627754    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:57.460100 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:57.460065    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:58.366838 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:58.366808    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:04:58.994663 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:04:58.994626    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:00.275759 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:00.275723    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 19 12:05:00.276241 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:00.275782    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95" podUID="7bd4a0cb-4a14-4382-895b-9af64aad5f72" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 12:05:01.361729 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.361654    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:05:01.401715 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.401599    2584 generic.go:356] "Generic (PLEG): container finished" podID="7bd4a0cb-4a14-4382-895b-9af64aad5f72" containerID="ca05bdc1231d29cf7ca707632c18a156bf61b07f3b05fa31477b541c7f354551" exitCode=0
Sep 19 12:05:01.401715 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.401666    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"
Sep 19 12:05:01.401715 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.401674    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95" event={"ID":"7bd4a0cb-4a14-4382-895b-9af64aad5f72","Type":"ContainerDied","Data":"ca05bdc1231d29cf7ca707632c18a156bf61b07f3b05fa31477b541c7f354551"}
Sep 19 12:05:01.402070 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.401729    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95" event={"ID":"7bd4a0cb-4a14-4382-895b-9af64aad5f72","Type":"ContainerDied","Data":"1f850343f400753e10a0af87d583b919aa2b2d15050a3bb4faa0825ab562d46e"}
Sep 19 12:05:01.402070 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.401761    2584 scope.go:117] "RemoveContainer" containerID="ca05bdc1231d29cf7ca707632c18a156bf61b07f3b05fa31477b541c7f354551"
Sep 19 12:05:01.410723 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.410698    2584 scope.go:117] "RemoveContainer" containerID="3b608c4bd64804fc1d785d3aab2c0695f32d9b24ff7999a42acd04054155b891"
Sep 19 12:05:01.416023 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.415999    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-data\") pod \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") "
Sep 19 12:05:01.416128 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.416032    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-workload-certs\") pod \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") "
Sep 19 12:05:01.416128 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.416063    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-podinfo\") pod \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") "
Sep 19 12:05:01.416128 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.416096    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istiod-ca-cert\") pod \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") "
Sep 19 12:05:01.416128 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.416116    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vz4r\" (UniqueName: \"kubernetes.io/projected/7bd4a0cb-4a14-4382-895b-9af64aad5f72-kube-api-access-7vz4r\") pod \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") "
Sep 19 12:05:01.416337 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.416273    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-token\") pod \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") "
Sep 19 12:05:01.416337 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.416288    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-data" (OuterVolumeSpecName: "istio-data") pod "7bd4a0cb-4a14-4382-895b-9af64aad5f72" (UID: "7bd4a0cb-4a14-4382-895b-9af64aad5f72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:05:01.416337 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.416313    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7bd4a0cb-4a14-4382-895b-9af64aad5f72" (UID: "7bd4a0cb-4a14-4382-895b-9af64aad5f72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:05:01.416337 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.416333    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-workload-socket\") pod \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") "
Sep 19 12:05:01.416558 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.416351    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-envoy\") pod \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") "
Sep 19 12:05:01.416558 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.416407    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-credential-socket\") pod \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\" (UID: \"7bd4a0cb-4a14-4382-895b-9af64aad5f72\") "
Sep 19 12:05:01.416663 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.416600    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7bd4a0cb-4a14-4382-895b-9af64aad5f72" (UID: "7bd4a0cb-4a14-4382-895b-9af64aad5f72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 12:05:01.416720 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.416702    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-data\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:05:01.416772 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.416720    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-workload-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:05:01.416843 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.416813    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7bd4a0cb-4a14-4382-895b-9af64aad5f72" (UID: "7bd4a0cb-4a14-4382-895b-9af64aad5f72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:05:01.416843 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.416835    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istiod-ca-cert\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:05:01.417082 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.417047    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7bd4a0cb-4a14-4382-895b-9af64aad5f72" (UID: "7bd4a0cb-4a14-4382-895b-9af64aad5f72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:05:01.418688 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.418662    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7bd4a0cb-4a14-4382-895b-9af64aad5f72" (UID: "7bd4a0cb-4a14-4382-895b-9af64aad5f72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 12:05:01.419111 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.419090    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-token" (OuterVolumeSpecName: "istio-token") pod "7bd4a0cb-4a14-4382-895b-9af64aad5f72" (UID: "7bd4a0cb-4a14-4382-895b-9af64aad5f72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 12:05:01.419204 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.419170    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd4a0cb-4a14-4382-895b-9af64aad5f72-kube-api-access-7vz4r" (OuterVolumeSpecName: "kube-api-access-7vz4r") pod "7bd4a0cb-4a14-4382-895b-9af64aad5f72" (UID: "7bd4a0cb-4a14-4382-895b-9af64aad5f72"). InnerVolumeSpecName "kube-api-access-7vz4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 12:05:01.419253 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.419228    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7bd4a0cb-4a14-4382-895b-9af64aad5f72" (UID: "7bd4a0cb-4a14-4382-895b-9af64aad5f72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:05:01.420113 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.420094    2584 scope.go:117] "RemoveContainer" containerID="ca05bdc1231d29cf7ca707632c18a156bf61b07f3b05fa31477b541c7f354551"
Sep 19 12:05:01.420441 ip-10-0-133-209 kubenswrapper[2584]: E0919 12:05:01.420421    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca05bdc1231d29cf7ca707632c18a156bf61b07f3b05fa31477b541c7f354551\": container with ID starting with ca05bdc1231d29cf7ca707632c18a156bf61b07f3b05fa31477b541c7f354551 not found: ID does not exist" containerID="ca05bdc1231d29cf7ca707632c18a156bf61b07f3b05fa31477b541c7f354551"
Sep 19 12:05:01.420487 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.420449    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca05bdc1231d29cf7ca707632c18a156bf61b07f3b05fa31477b541c7f354551"} err="failed to get container status \"ca05bdc1231d29cf7ca707632c18a156bf61b07f3b05fa31477b541c7f354551\": rpc error: code = NotFound desc = could not find container \"ca05bdc1231d29cf7ca707632c18a156bf61b07f3b05fa31477b541c7f354551\": container with ID starting with ca05bdc1231d29cf7ca707632c18a156bf61b07f3b05fa31477b541c7f354551 not found: ID does not exist"
Sep 19 12:05:01.420487 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.420466    2584 scope.go:117] "RemoveContainer" containerID="3b608c4bd64804fc1d785d3aab2c0695f32d9b24ff7999a42acd04054155b891"
Sep 19 12:05:01.420756 ip-10-0-133-209 kubenswrapper[2584]: E0919 12:05:01.420740    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b608c4bd64804fc1d785d3aab2c0695f32d9b24ff7999a42acd04054155b891\": container with ID starting with 3b608c4bd64804fc1d785d3aab2c0695f32d9b24ff7999a42acd04054155b891 not found: ID does not exist" containerID="3b608c4bd64804fc1d785d3aab2c0695f32d9b24ff7999a42acd04054155b891"
Sep 19 12:05:01.420815 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.420763    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b608c4bd64804fc1d785d3aab2c0695f32d9b24ff7999a42acd04054155b891"} err="failed to get container status \"3b608c4bd64804fc1d785d3aab2c0695f32d9b24ff7999a42acd04054155b891\": rpc error: code = NotFound desc = could not find container \"3b608c4bd64804fc1d785d3aab2c0695f32d9b24ff7999a42acd04054155b891\": container with ID starting with 3b608c4bd64804fc1d785d3aab2c0695f32d9b24ff7999a42acd04054155b891 not found: ID does not exist"
Sep 19 12:05:01.517944 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.517871    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-podinfo\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:05:01.517944 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.517905    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vz4r\" (UniqueName: \"kubernetes.io/projected/7bd4a0cb-4a14-4382-895b-9af64aad5f72-kube-api-access-7vz4r\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:05:01.517944 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.517951    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-token\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:05:01.518191 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.517963    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-workload-socket\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:05:01.518191 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.517974    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-istio-envoy\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:05:01.518191 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.517985    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd4a0cb-4a14-4382-895b-9af64aad5f72-credential-socket\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:05:01.724613 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.724577    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"]
Sep 19 12:05:01.728200 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.728154    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-df64a0-openshift-default-6d854dc94b-skv95"]
Sep 19 12:05:01.904619 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:01.904582    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bd4a0cb-4a14-4382-895b-9af64aad5f72" path="/var/lib/kubelet/pods/7bd4a0cb-4a14-4382-895b-9af64aad5f72/volumes"
Sep 19 12:05:07.071405 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:07.071368    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4937ff-29830320-svghl"]
Sep 19 12:05:07.074115 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:07.074084    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4937ff-29830320-svghl"]
Sep 19 12:05:07.076114 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:07.076085    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:07.134455 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:07.134428    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:07.234679 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:07.234652    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:07.286468 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:07.286427    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:07.622297 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:07.622254    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:07.721343 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:07.721305    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:07.906050 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:07.905964    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e583e2a-2f1f-41a6-bbd8-2430b405bb3d" path="/var/lib/kubelet/pods/1e583e2a-2f1f-41a6-bbd8-2430b405bb3d/volumes"
Sep 19 12:05:08.354125 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:08.354093    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:13.474374 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:13.474317    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:13.534122 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:13.534091    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:13.716078 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:13.716035    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:14.340822 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:14.340784    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:14.974017 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:14.973985    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:15.763592 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:15.763559    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:24.520693 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:24.520649    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"]
Sep 19 12:05:24.521342 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:24.520884    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q" podUID="f967afc0-b956-4227-97fe-a1329c2e544a" containerName="maas-api" containerID="cri-o://d5fddbf886e9e2e38c1758028a67401b76858ea3553365c6619a3977ef39f01e" gracePeriod=30
Sep 19 12:05:24.547189 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:24.547157    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:24.648462 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:24.648425    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:24.723113 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:24.723083    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:24.777581 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:24.777466    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:24.777760 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:24.777693    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"
Sep 19 12:05:24.936585 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:24.936543    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f967afc0-b956-4227-97fe-a1329c2e544a-maas-api-tls\") pod \"f967afc0-b956-4227-97fe-a1329c2e544a\" (UID: \"f967afc0-b956-4227-97fe-a1329c2e544a\") "
Sep 19 12:05:24.936796 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:24.936611    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f967afc0-b956-4227-97fe-a1329c2e544a-metrics-certs\") pod \"f967afc0-b956-4227-97fe-a1329c2e544a\" (UID: \"f967afc0-b956-4227-97fe-a1329c2e544a\") "
Sep 19 12:05:24.936796 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:24.936649    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkcdq\" (UniqueName: \"kubernetes.io/projected/f967afc0-b956-4227-97fe-a1329c2e544a-kube-api-access-tkcdq\") pod \"f967afc0-b956-4227-97fe-a1329c2e544a\" (UID: \"f967afc0-b956-4227-97fe-a1329c2e544a\") "
Sep 19 12:05:24.936796 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:24.936674    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f967afc0-b956-4227-97fe-a1329c2e544a-openshift-service-ca\") pod \"f967afc0-b956-4227-97fe-a1329c2e544a\" (UID: \"f967afc0-b956-4227-97fe-a1329c2e544a\") "
Sep 19 12:05:24.937168 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:24.937135    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f967afc0-b956-4227-97fe-a1329c2e544a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f967afc0-b956-4227-97fe-a1329c2e544a" (UID: "f967afc0-b956-4227-97fe-a1329c2e544a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 12:05:24.939264 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:24.939224    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f967afc0-b956-4227-97fe-a1329c2e544a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f967afc0-b956-4227-97fe-a1329c2e544a" (UID: "f967afc0-b956-4227-97fe-a1329c2e544a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 12:05:24.939264 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:24.939231    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f967afc0-b956-4227-97fe-a1329c2e544a-kube-api-access-tkcdq" (OuterVolumeSpecName: "kube-api-access-tkcdq") pod "f967afc0-b956-4227-97fe-a1329c2e544a" (UID: "f967afc0-b956-4227-97fe-a1329c2e544a"). InnerVolumeSpecName "kube-api-access-tkcdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 12:05:24.939421 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:24.939296    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f967afc0-b956-4227-97fe-a1329c2e544a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f967afc0-b956-4227-97fe-a1329c2e544a" (UID: "f967afc0-b956-4227-97fe-a1329c2e544a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 12:05:25.038194 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:25.038065    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f967afc0-b956-4227-97fe-a1329c2e544a-metrics-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:05:25.038194 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:25.038123    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkcdq\" (UniqueName: \"kubernetes.io/projected/f967afc0-b956-4227-97fe-a1329c2e544a-kube-api-access-tkcdq\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:05:25.038194 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:25.038135    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f967afc0-b956-4227-97fe-a1329c2e544a-openshift-service-ca\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:05:25.038194 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:25.038149    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f967afc0-b956-4227-97fe-a1329c2e544a-maas-api-tls\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:05:25.115169 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:25.115139    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:25.204625 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:25.204592    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:25.536110 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:25.536073    2584 generic.go:356] "Generic (PLEG): container finished" podID="f967afc0-b956-4227-97fe-a1329c2e544a" containerID="d5fddbf886e9e2e38c1758028a67401b76858ea3553365c6619a3977ef39f01e" exitCode=0
Sep 19 12:05:25.536607 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:25.536120    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"
Sep 19 12:05:25.536607 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:25.536138    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q" event={"ID":"f967afc0-b956-4227-97fe-a1329c2e544a","Type":"ContainerDied","Data":"d5fddbf886e9e2e38c1758028a67401b76858ea3553365c6619a3977ef39f01e"}
Sep 19 12:05:25.536607 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:25.536182    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q" event={"ID":"f967afc0-b956-4227-97fe-a1329c2e544a","Type":"ContainerDied","Data":"3f8f0650eb2c86f70182734091ba0b8a28e4b2b05c18f3f918757c0f4e5f3153"}
Sep 19 12:05:25.536607 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:25.536204    2584 scope.go:117] "RemoveContainer" containerID="d5fddbf886e9e2e38c1758028a67401b76858ea3553365c6619a3977ef39f01e"
Sep 19 12:05:25.546042 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:25.546021    2584 scope.go:117] "RemoveContainer" containerID="d5fddbf886e9e2e38c1758028a67401b76858ea3553365c6619a3977ef39f01e"
Sep 19 12:05:25.546431 ip-10-0-133-209 kubenswrapper[2584]: E0919 12:05:25.546363    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5fddbf886e9e2e38c1758028a67401b76858ea3553365c6619a3977ef39f01e\": container with ID starting with d5fddbf886e9e2e38c1758028a67401b76858ea3553365c6619a3977ef39f01e not found: ID does not exist" containerID="d5fddbf886e9e2e38c1758028a67401b76858ea3553365c6619a3977ef39f01e"
Sep 19 12:05:25.546431 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:25.546404    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5fddbf886e9e2e38c1758028a67401b76858ea3553365c6619a3977ef39f01e"} err="failed to get container status \"d5fddbf886e9e2e38c1758028a67401b76858ea3553365c6619a3977ef39f01e\": rpc error: code = NotFound desc = could not find container \"d5fddbf886e9e2e38c1758028a67401b76858ea3553365c6619a3977ef39f01e\": container with ID starting with d5fddbf886e9e2e38c1758028a67401b76858ea3553365c6619a3977ef39f01e not found: ID does not exist"
Sep 19 12:05:25.557905 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:25.557859    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"]
Sep 19 12:05:25.564937 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:25.564891    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4a8a59-5b5b6dcb55-xsk6q"]
Sep 19 12:05:25.904267 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:25.904230    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f967afc0-b956-4227-97fe-a1329c2e544a" path="/var/lib/kubelet/pods/f967afc0-b956-4227-97fe-a1329c2e544a/volumes"
Sep 19 12:05:30.873654 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:30.873610    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:30.929173 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:30.929140    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:05:31.137386 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:05:31.137290    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:06:00.970670 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:06:00.970630    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:06:32.909379 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:06:32.909343    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:06:32.959546 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:06:32.959514    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:06:38.264070 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:06:38.264033    2584 scope.go:117] "RemoveContainer" containerID="478781711d006fcc1615d0bbadbe0c82fa04768592b324c882227db11347a419"
Sep 19 12:06:58.793239 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:06:58.793202    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:07:00.570118 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:07:00.570080    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:07:22.727856 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:07:22.727827    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:07:22.827009 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:07:22.826976    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:07:48.971193 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:07:48.971106    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:07:49.054664 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:07:49.054627    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:08:38.034082 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:08:38.034050    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/1.log"
Sep 19 12:08:38.039344 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:08:38.039321    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/1.log"
Sep 19 12:08:38.040358 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:08:38.040333    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6th98_2fed27e0-833f-49a8-b11e-8f61df6e6bd5/ovn-acl-logging/0.log"
Sep 19 12:08:38.045497 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:08:38.045476    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6th98_2fed27e0-833f-49a8-b11e-8f61df6e6bd5/ovn-acl-logging/0.log"
Sep 19 12:09:06.999189 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:06.999152    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"]
Sep 19 12:09:06.999763 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:06.999401    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg" podUID="9b233060-ef58-4c7c-a92c-a39ffd617014" containerName="manager" containerID="cri-o://309bc0daeb4a81484072fd606af98bf4e77ce7fd677fcb1ae5d63e8e2ac58021" gracePeriod=10
Sep 19 12:09:08.879070 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:08.879039    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:09:09.060374 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.060348    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"
Sep 19 12:09:09.232820 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.232723    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b233060-ef58-4c7c-a92c-a39ffd617014-extensions-socket-volume\") pod \"9b233060-ef58-4c7c-a92c-a39ffd617014\" (UID: \"9b233060-ef58-4c7c-a92c-a39ffd617014\") "
Sep 19 12:09:09.232820 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.232790    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw68t\" (UniqueName: \"kubernetes.io/projected/9b233060-ef58-4c7c-a92c-a39ffd617014-kube-api-access-mw68t\") pod \"9b233060-ef58-4c7c-a92c-a39ffd617014\" (UID: \"9b233060-ef58-4c7c-a92c-a39ffd617014\") "
Sep 19 12:09:09.233190 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.233163    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b233060-ef58-4c7c-a92c-a39ffd617014-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9b233060-ef58-4c7c-a92c-a39ffd617014" (UID: "9b233060-ef58-4c7c-a92c-a39ffd617014"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:09:09.235184 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.235159    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b233060-ef58-4c7c-a92c-a39ffd617014-kube-api-access-mw68t" (OuterVolumeSpecName: "kube-api-access-mw68t") pod "9b233060-ef58-4c7c-a92c-a39ffd617014" (UID: "9b233060-ef58-4c7c-a92c-a39ffd617014"). InnerVolumeSpecName "kube-api-access-mw68t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 12:09:09.334094 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.334048    2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b233060-ef58-4c7c-a92c-a39ffd617014-extensions-socket-volume\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:09:09.334094 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.334084    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw68t\" (UniqueName: \"kubernetes.io/projected/9b233060-ef58-4c7c-a92c-a39ffd617014-kube-api-access-mw68t\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:09:09.675631 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.675593    2584 generic.go:356] "Generic (PLEG): container finished" podID="9b233060-ef58-4c7c-a92c-a39ffd617014" containerID="309bc0daeb4a81484072fd606af98bf4e77ce7fd677fcb1ae5d63e8e2ac58021" exitCode=0
Sep 19 12:09:09.675842 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.675640    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"
Sep 19 12:09:09.675842 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.675656    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg" event={"ID":"9b233060-ef58-4c7c-a92c-a39ffd617014","Type":"ContainerDied","Data":"309bc0daeb4a81484072fd606af98bf4e77ce7fd677fcb1ae5d63e8e2ac58021"}
Sep 19 12:09:09.675842 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.675708    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg" event={"ID":"9b233060-ef58-4c7c-a92c-a39ffd617014","Type":"ContainerDied","Data":"689aef5bdd195b03779180294681058486d43bdeed48075ac5d38c3cbf8d643b"}
Sep 19 12:09:09.675842 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.675737    2584 scope.go:117] "RemoveContainer" containerID="309bc0daeb4a81484072fd606af98bf4e77ce7fd677fcb1ae5d63e8e2ac58021"
Sep 19 12:09:09.687129 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.687102    2584 scope.go:117] "RemoveContainer" containerID="a5938c0b26a0a00808b5f4cab92caa38017a11e3c886cba25311e42acfe5d592"
Sep 19 12:09:09.696696 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.696668    2584 scope.go:117] "RemoveContainer" containerID="309bc0daeb4a81484072fd606af98bf4e77ce7fd677fcb1ae5d63e8e2ac58021"
Sep 19 12:09:09.697187 ip-10-0-133-209 kubenswrapper[2584]: E0919 12:09:09.697158    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"309bc0daeb4a81484072fd606af98bf4e77ce7fd677fcb1ae5d63e8e2ac58021\": container with ID starting with 309bc0daeb4a81484072fd606af98bf4e77ce7fd677fcb1ae5d63e8e2ac58021 not found: ID does not exist" containerID="309bc0daeb4a81484072fd606af98bf4e77ce7fd677fcb1ae5d63e8e2ac58021"
Sep 19 12:09:09.697294 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.697201    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"309bc0daeb4a81484072fd606af98bf4e77ce7fd677fcb1ae5d63e8e2ac58021"} err="failed to get container status \"309bc0daeb4a81484072fd606af98bf4e77ce7fd677fcb1ae5d63e8e2ac58021\": rpc error: code = NotFound desc = could not find container \"309bc0daeb4a81484072fd606af98bf4e77ce7fd677fcb1ae5d63e8e2ac58021\": container with ID starting with 309bc0daeb4a81484072fd606af98bf4e77ce7fd677fcb1ae5d63e8e2ac58021 not found: ID does not exist"
Sep 19 12:09:09.697294 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.697230    2584 scope.go:117] "RemoveContainer" containerID="a5938c0b26a0a00808b5f4cab92caa38017a11e3c886cba25311e42acfe5d592"
Sep 19 12:09:09.697571 ip-10-0-133-209 kubenswrapper[2584]: E0919 12:09:09.697548    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5938c0b26a0a00808b5f4cab92caa38017a11e3c886cba25311e42acfe5d592\": container with ID starting with a5938c0b26a0a00808b5f4cab92caa38017a11e3c886cba25311e42acfe5d592 not found: ID does not exist" containerID="a5938c0b26a0a00808b5f4cab92caa38017a11e3c886cba25311e42acfe5d592"
Sep 19 12:09:09.697665 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.697575    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5938c0b26a0a00808b5f4cab92caa38017a11e3c886cba25311e42acfe5d592"} err="failed to get container status \"a5938c0b26a0a00808b5f4cab92caa38017a11e3c886cba25311e42acfe5d592\": rpc error: code = NotFound desc = could not find container \"a5938c0b26a0a00808b5f4cab92caa38017a11e3c886cba25311e42acfe5d592\": container with ID starting with a5938c0b26a0a00808b5f4cab92caa38017a11e3c886cba25311e42acfe5d592 not found: ID does not exist"
Sep 19 12:09:09.697745 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.697690    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"]
Sep 19 12:09:09.701365 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.701337    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-k7fwg"]
Sep 19 12:09:09.902774 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:09:09.902740    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b233060-ef58-4c7c-a92c-a39ffd617014" path="/var/lib/kubelet/pods/9b233060-ef58-4c7c-a92c-a39ffd617014/volumes"
Sep 19 12:10:07.683723 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.683688    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-9kz6w"]
Sep 19 12:10:07.684225 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684052    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82514835-6ded-4b36-96b1-6e260624de3f" containerName="istio-proxy"
Sep 19 12:10:07.684225 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684066    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="82514835-6ded-4b36-96b1-6e260624de3f" containerName="istio-proxy"
Sep 19 12:10:07.684225 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684077    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd4a0cb-4a14-4382-895b-9af64aad5f72" containerName="istio-proxy"
Sep 19 12:10:07.684225 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684083    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd4a0cb-4a14-4382-895b-9af64aad5f72" containerName="istio-proxy"
Sep 19 12:10:07.684225 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684091    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6373b046-ef34-41d4-ac1e-afd01944cd12" containerName="istio-proxy"
Sep 19 12:10:07.684225 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684097    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6373b046-ef34-41d4-ac1e-afd01944cd12" containerName="istio-proxy"
Sep 19 12:10:07.684225 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684110    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b233060-ef58-4c7c-a92c-a39ffd617014" containerName="manager"
Sep 19 12:10:07.684225 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684115    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b233060-ef58-4c7c-a92c-a39ffd617014" containerName="manager"
Sep 19 12:10:07.684225 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684124    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b444e909-bc58-42b7-9405-840b39e3584d" containerName="main"
Sep 19 12:10:07.684225 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684130    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b444e909-bc58-42b7-9405-840b39e3584d" containerName="main"
Sep 19 12:10:07.684225 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684135    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b233060-ef58-4c7c-a92c-a39ffd617014" containerName="manager"
Sep 19 12:10:07.684225 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684140    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b233060-ef58-4c7c-a92c-a39ffd617014" containerName="manager"
Sep 19 12:10:07.684225 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684146    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82514835-6ded-4b36-96b1-6e260624de3f" containerName="istio-proxy"
Sep 19 12:10:07.684225 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684150    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="82514835-6ded-4b36-96b1-6e260624de3f" containerName="istio-proxy"
Sep 19 12:10:07.684225 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684164    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f967afc0-b956-4227-97fe-a1329c2e544a" containerName="maas-api"
Sep 19 12:10:07.684225 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684169    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f967afc0-b956-4227-97fe-a1329c2e544a" containerName="maas-api"
Sep 19 12:10:07.684225 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684175    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49132f35-53b0-42fb-977c-f0566ce60304" containerName="maas-api"
Sep 19 12:10:07.684225 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684180    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="49132f35-53b0-42fb-977c-f0566ce60304" containerName="maas-api"
Sep 19 12:10:07.684225 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684234    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f967afc0-b956-4227-97fe-a1329c2e544a" containerName="maas-api"
Sep 19 12:10:07.684881 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684241    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6373b046-ef34-41d4-ac1e-afd01944cd12" containerName="istio-proxy"
Sep 19 12:10:07.684881 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684249    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="49132f35-53b0-42fb-977c-f0566ce60304" containerName="maas-api"
Sep 19 12:10:07.684881 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684254    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd4a0cb-4a14-4382-895b-9af64aad5f72" containerName="istio-proxy"
Sep 19 12:10:07.684881 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684260    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b233060-ef58-4c7c-a92c-a39ffd617014" containerName="manager"
Sep 19 12:10:07.684881 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684265    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="82514835-6ded-4b36-96b1-6e260624de3f" containerName="istio-proxy"
Sep 19 12:10:07.684881 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684270    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b233060-ef58-4c7c-a92c-a39ffd617014" containerName="manager"
Sep 19 12:10:07.684881 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684277    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd4a0cb-4a14-4382-895b-9af64aad5f72" containerName="istio-proxy"
Sep 19 12:10:07.684881 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684284    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="82514835-6ded-4b36-96b1-6e260624de3f" containerName="istio-proxy"
Sep 19 12:10:07.684881 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.684290    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b444e909-bc58-42b7-9405-840b39e3584d" containerName="main"
Sep 19 12:10:07.687274 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.687252    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-9kz6w"
Sep 19 12:10:07.690141 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.690116    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8nxp4\""
Sep 19 12:10:07.698077 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.698047    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-9kz6w"]
Sep 19 12:10:07.750177 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.750128    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zc6p\" (UniqueName: \"kubernetes.io/projected/b90272ae-624f-4c2f-a9f0-aee7c565f4f3-kube-api-access-9zc6p\") pod \"kuadrant-operator-controller-manager-54b6d5c989-9kz6w\" (UID: \"b90272ae-624f-4c2f-a9f0-aee7c565f4f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-9kz6w"
Sep 19 12:10:07.750177 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.750177    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b90272ae-624f-4c2f-a9f0-aee7c565f4f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-54b6d5c989-9kz6w\" (UID: \"b90272ae-624f-4c2f-a9f0-aee7c565f4f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-9kz6w"
Sep 19 12:10:07.851555 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.851508    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zc6p\" (UniqueName: \"kubernetes.io/projected/b90272ae-624f-4c2f-a9f0-aee7c565f4f3-kube-api-access-9zc6p\") pod \"kuadrant-operator-controller-manager-54b6d5c989-9kz6w\" (UID: \"b90272ae-624f-4c2f-a9f0-aee7c565f4f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-9kz6w"
Sep 19 12:10:07.851555 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.851554    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b90272ae-624f-4c2f-a9f0-aee7c565f4f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-54b6d5c989-9kz6w\" (UID: \"b90272ae-624f-4c2f-a9f0-aee7c565f4f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-9kz6w"
Sep 19 12:10:07.851957 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.851907    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b90272ae-624f-4c2f-a9f0-aee7c565f4f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-54b6d5c989-9kz6w\" (UID: \"b90272ae-624f-4c2f-a9f0-aee7c565f4f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-9kz6w"
Sep 19 12:10:07.859962 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:07.859904    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zc6p\" (UniqueName: \"kubernetes.io/projected/b90272ae-624f-4c2f-a9f0-aee7c565f4f3-kube-api-access-9zc6p\") pod \"kuadrant-operator-controller-manager-54b6d5c989-9kz6w\" (UID: \"b90272ae-624f-4c2f-a9f0-aee7c565f4f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-9kz6w"
Sep 19 12:10:08.000929 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:08.000812    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-9kz6w"
Sep 19 12:10:08.278596 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:08.278542    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-9kz6w"]
Sep 19 12:10:08.281727 ip-10-0-133-209 kubenswrapper[2584]: W0919 12:10:08.281698    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90272ae_624f_4c2f_a9f0_aee7c565f4f3.slice/crio-388fd9de4fe73a31ecf699e2b4e2d6d1d3c3939d72075672c907b12390026d45 WatchSource:0}: Error finding container 388fd9de4fe73a31ecf699e2b4e2d6d1d3c3939d72075672c907b12390026d45: Status 404 returned error can't find the container with id 388fd9de4fe73a31ecf699e2b4e2d6d1d3c3939d72075672c907b12390026d45
Sep 19 12:10:08.973550 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:08.973513    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-9kz6w" event={"ID":"b90272ae-624f-4c2f-a9f0-aee7c565f4f3","Type":"ContainerStarted","Data":"972c73be8c5dfd00b53dae9cd78668aa270702201a4b3a9fbd9be239c00e5dfc"}
Sep 19 12:10:08.974015 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:08.973569    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-9kz6w" event={"ID":"b90272ae-624f-4c2f-a9f0-aee7c565f4f3","Type":"ContainerStarted","Data":"388fd9de4fe73a31ecf699e2b4e2d6d1d3c3939d72075672c907b12390026d45"}
Sep 19 12:10:08.974015 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:08.973697    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-9kz6w"
Sep 19 12:10:08.995652 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:08.995591    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-9kz6w" podStartSLOduration=1.995573375 podStartE2EDuration="1.995573375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:10:08.992353266 +0000 UTC m=+2193.192383070" watchObservedRunningTime="2026-09-19 12:10:08.995573375 +0000 UTC m=+2193.195603178"
Sep 19 12:10:19.981503 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:19.981471    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-54b6d5c989-9kz6w"
Sep 19 12:10:56.943898 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:56.943802    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"]
Sep 19 12:10:56.944528 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:56.944255    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd4a0cb-4a14-4382-895b-9af64aad5f72" containerName="istio-proxy"
Sep 19 12:10:56.944528 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:56.944268    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd4a0cb-4a14-4382-895b-9af64aad5f72" containerName="istio-proxy"
Sep 19 12:10:56.944528 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:56.944277    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6373b046-ef34-41d4-ac1e-afd01944cd12" containerName="istio-proxy"
Sep 19 12:10:56.944528 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:56.944282    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6373b046-ef34-41d4-ac1e-afd01944cd12" containerName="istio-proxy"
Sep 19 12:10:56.944528 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:56.944352    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6373b046-ef34-41d4-ac1e-afd01944cd12" containerName="istio-proxy"
Sep 19 12:10:56.947389 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:56.947368    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:56.950072 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:56.950043    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-3f116f14-kserve-self-signed-certs\""
Sep 19 12:10:56.955861 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:56.955837    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"]
Sep 19 12:10:56.992459 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:56.992417    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-model-cache\") pod \"e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") " pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:56.992663 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:56.992492    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swjx5\" (UniqueName: \"kubernetes.io/projected/4272a283-8833-42dc-b049-dd3f0f70515c-kube-api-access-swjx5\") pod \"e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") " pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:56.992663 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:56.992557    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-dshm\") pod \"e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") " pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:56.992663 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:56.992586    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4272a283-8833-42dc-b049-dd3f0f70515c-tls-certs\") pod \"e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") " pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:56.992663 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:56.992655    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-tmp-dir\") pod \"e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") " pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:56.992855 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:56.992694    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-home\") pod \"e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") " pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:56.993178 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:56.993149    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:10:57.093542 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:57.093503    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-model-cache\") pod \"e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") " pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:57.093542 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:57.093561    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swjx5\" (UniqueName: \"kubernetes.io/projected/4272a283-8833-42dc-b049-dd3f0f70515c-kube-api-access-swjx5\") pod \"e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") " pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:57.093860 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:57.093596    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-dshm\") pod \"e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") " pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:57.093860 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:57.093627    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4272a283-8833-42dc-b049-dd3f0f70515c-tls-certs\") pod \"e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") " pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:57.093860 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:57.093679    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-tmp-dir\") pod \"e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") " pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:57.093860 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:57.093717    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-home\") pod \"e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") " pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:57.094150 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:57.094055    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-model-cache\") pod \"e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") " pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:57.094257 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:57.094226    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-tmp-dir\") pod \"e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") " pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:57.094452 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:57.094432    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-home\") pod \"e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") " pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:57.096505 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:57.096478    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-dshm\") pod \"e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") " pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:57.096605 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:57.096537    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4272a283-8833-42dc-b049-dd3f0f70515c-tls-certs\") pod \"e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") " pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:57.101796 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:57.101768    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swjx5\" (UniqueName: \"kubernetes.io/projected/4272a283-8833-42dc-b049-dd3f0f70515c-kube-api-access-swjx5\") pod \"e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") " pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:57.258171 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:57.258072    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:10:57.532029 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:57.532004    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"]
Sep 19 12:10:57.534116 ip-10-0-133-209 kubenswrapper[2584]: W0919 12:10:57.534083    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4272a283_8833_42dc_b049_dd3f0f70515c.slice/crio-bde29736c94ada465c6c3c373be6ec609a3f725963b444da770ba8286a8789c8 WatchSource:0}: Error finding container bde29736c94ada465c6c3c373be6ec609a3f725963b444da770ba8286a8789c8: Status 404 returned error can't find the container with id bde29736c94ada465c6c3c373be6ec609a3f725963b444da770ba8286a8789c8
Sep 19 12:10:58.218203 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:58.218162    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh" event={"ID":"4272a283-8833-42dc-b049-dd3f0f70515c","Type":"ContainerStarted","Data":"3f3555a1375c05b01f05e1670ca181f91e7bad69331af3e15f4be16836b28725"}
Sep 19 12:10:58.218203 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:58.218205    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh" event={"ID":"4272a283-8833-42dc-b049-dd3f0f70515c","Type":"ContainerStarted","Data":"bde29736c94ada465c6c3c373be6ec609a3f725963b444da770ba8286a8789c8"}
Sep 19 12:10:58.236282 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:58.236226    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh" podStartSLOduration=2.236210459 podStartE2EDuration="2.236210459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:10:58.235384022 +0000 UTC m=+2242.435413827" watchObservedRunningTime="2026-09-19 12:10:58.236210459 +0000 UTC m=+2242.436240261"
Sep 19 12:10:58.258535 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:10:58.258494    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:11:07.258798 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:07.258762    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:11:07.271685 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:07.271655    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:11:07.282928 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:07.282883    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:11:16.066626 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:16.066583    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:11:18.672674 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:18.672638    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:11:28.562492 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:28.562459    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"]
Sep 19 12:11:28.571070 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:28.571038    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:11:31.095904 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:31.095869    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:11:43.584697 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:43.584629    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh" podUID="4272a283-8833-42dc-b049-dd3f0f70515c" containerName="main" containerID="cri-o://3f3555a1375c05b01f05e1670ca181f91e7bad69331af3e15f4be16836b28725" gracePeriod=45
Sep 19 12:11:43.836152 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:43.836084    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:11:43.929749 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:43.929711    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-tmp-dir\") pod \"4272a283-8833-42dc-b049-dd3f0f70515c\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") "
Sep 19 12:11:43.929993 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:43.929795    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-home\") pod \"4272a283-8833-42dc-b049-dd3f0f70515c\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") "
Sep 19 12:11:43.929993 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:43.929827    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-model-cache\") pod \"4272a283-8833-42dc-b049-dd3f0f70515c\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") "
Sep 19 12:11:43.929993 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:43.929846    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4272a283-8833-42dc-b049-dd3f0f70515c-tls-certs\") pod \"4272a283-8833-42dc-b049-dd3f0f70515c\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") "
Sep 19 12:11:43.929993 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:43.929862    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swjx5\" (UniqueName: \"kubernetes.io/projected/4272a283-8833-42dc-b049-dd3f0f70515c-kube-api-access-swjx5\") pod \"4272a283-8833-42dc-b049-dd3f0f70515c\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") "
Sep 19 12:11:43.929993 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:43.929879    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-dshm\") pod \"4272a283-8833-42dc-b049-dd3f0f70515c\" (UID: \"4272a283-8833-42dc-b049-dd3f0f70515c\") "
Sep 19 12:11:43.930285 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:43.930077    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4272a283-8833-42dc-b049-dd3f0f70515c" (UID: "4272a283-8833-42dc-b049-dd3f0f70515c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:11:43.930285 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:43.930176    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-home" (OuterVolumeSpecName: "home") pod "4272a283-8833-42dc-b049-dd3f0f70515c" (UID: "4272a283-8833-42dc-b049-dd3f0f70515c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:11:43.930285 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:43.930242    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-tmp-dir\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:11:43.930285 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:43.930262    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-home\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:11:43.930480 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:43.930402    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4272a283-8833-42dc-b049-dd3f0f70515c" (UID: "4272a283-8833-42dc-b049-dd3f0f70515c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:11:43.932530 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:43.932506    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4272a283-8833-42dc-b049-dd3f0f70515c-kube-api-access-swjx5" (OuterVolumeSpecName: "kube-api-access-swjx5") pod "4272a283-8833-42dc-b049-dd3f0f70515c" (UID: "4272a283-8833-42dc-b049-dd3f0f70515c"). InnerVolumeSpecName "kube-api-access-swjx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 12:11:43.932779 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:43.932754    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4272a283-8833-42dc-b049-dd3f0f70515c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4272a283-8833-42dc-b049-dd3f0f70515c" (UID: "4272a283-8833-42dc-b049-dd3f0f70515c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 12:11:43.932878 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:43.932777    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-dshm" (OuterVolumeSpecName: "dshm") pod "4272a283-8833-42dc-b049-dd3f0f70515c" (UID: "4272a283-8833-42dc-b049-dd3f0f70515c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 12:11:44.031346 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:44.031302    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-model-cache\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:11:44.031346 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:44.031338    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4272a283-8833-42dc-b049-dd3f0f70515c-tls-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:11:44.031346 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:44.031347    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swjx5\" (UniqueName: \"kubernetes.io/projected/4272a283-8833-42dc-b049-dd3f0f70515c-kube-api-access-swjx5\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:11:44.031346 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:44.031357    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4272a283-8833-42dc-b049-dd3f0f70515c-dshm\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:11:44.453374 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:44.453341    2584 generic.go:356] "Generic (PLEG): container finished" podID="4272a283-8833-42dc-b049-dd3f0f70515c" containerID="3f3555a1375c05b01f05e1670ca181f91e7bad69331af3e15f4be16836b28725" exitCode=0
Sep 19 12:11:44.453589 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:44.453389    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"
Sep 19 12:11:44.453589 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:44.453405    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh" event={"ID":"4272a283-8833-42dc-b049-dd3f0f70515c","Type":"ContainerDied","Data":"3f3555a1375c05b01f05e1670ca181f91e7bad69331af3e15f4be16836b28725"}
Sep 19 12:11:44.453589 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:44.453436    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh" event={"ID":"4272a283-8833-42dc-b049-dd3f0f70515c","Type":"ContainerDied","Data":"bde29736c94ada465c6c3c373be6ec609a3f725963b444da770ba8286a8789c8"}
Sep 19 12:11:44.453589 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:44.453452    2584 scope.go:117] "RemoveContainer" containerID="3f3555a1375c05b01f05e1670ca181f91e7bad69331af3e15f4be16836b28725"
Sep 19 12:11:44.462609 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:44.462589    2584 scope.go:117] "RemoveContainer" containerID="3f3555a1375c05b01f05e1670ca181f91e7bad69331af3e15f4be16836b28725"
Sep 19 12:11:44.462884 ip-10-0-133-209 kubenswrapper[2584]: E0919 12:11:44.462867    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f3555a1375c05b01f05e1670ca181f91e7bad69331af3e15f4be16836b28725\": container with ID starting with 3f3555a1375c05b01f05e1670ca181f91e7bad69331af3e15f4be16836b28725 not found: ID does not exist" containerID="3f3555a1375c05b01f05e1670ca181f91e7bad69331af3e15f4be16836b28725"
Sep 19 12:11:44.462994 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:44.462891    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3555a1375c05b01f05e1670ca181f91e7bad69331af3e15f4be16836b28725"} err="failed to get container status \"3f3555a1375c05b01f05e1670ca181f91e7bad69331af3e15f4be16836b28725\": rpc error: code = NotFound desc = could not find container \"3f3555a1375c05b01f05e1670ca181f91e7bad69331af3e15f4be16836b28725\": container with ID starting with 3f3555a1375c05b01f05e1670ca181f91e7bad69331af3e15f4be16836b28725 not found: ID does not exist"
Sep 19 12:11:44.473923 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:44.473886    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"]
Sep 19 12:11:44.477460 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:44.477437    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-3f116f14-kserve-5879999d76-f6sxh"]
Sep 19 12:11:45.903292 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:45.903254    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4272a283-8833-42dc-b049-dd3f0f70515c" path="/var/lib/kubelet/pods/4272a283-8833-42dc-b049-dd3f0f70515c/volumes"
Sep 19 12:11:58.533366 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.533333    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6649958464-nszbc"]
Sep 19 12:11:58.533776 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.533711    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4272a283-8833-42dc-b049-dd3f0f70515c" containerName="main"
Sep 19 12:11:58.533776 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.533722    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4272a283-8833-42dc-b049-dd3f0f70515c" containerName="main"
Sep 19 12:11:58.533889 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.533799    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4272a283-8833-42dc-b049-dd3f0f70515c" containerName="main"
Sep 19 12:11:58.538477 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.538450    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc"
Sep 19 12:11:58.541672 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.541633    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 19 12:11:58.541672 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.541640    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-g4wpl\""
Sep 19 12:11:58.541884 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.541754    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 19 12:11:58.541884 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.541640    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 12:11:58.542448 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.542432    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 19 12:11:58.542493 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.542457    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c"]
Sep 19 12:11:58.546605 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.546222    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6649958464-nszbc"]
Sep 19 12:11:58.546605 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.546327    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c"
Sep 19 12:11:58.558266 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.558243    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c"]
Sep 19 12:11:58.670325 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.670284    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f710bc83-dc05-4d3f-a314-f92271e2e8b9-openshift-service-ca\") pod \"maas-api-6649958464-nszbc\" (UID: \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\") " pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc"
Sep 19 12:11:58.670325 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.670327    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62-metrics-certs\") pod \"maas-api-7bd8599656-m4c6c\" (UID: \"b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62\") " pod="odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c"
Sep 19 12:11:58.670591 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.670352    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62-maas-api-tls\") pod \"maas-api-7bd8599656-m4c6c\" (UID: \"b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62\") " pod="odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c"
Sep 19 12:11:58.670591 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.670423    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hrn9\" (UniqueName: \"kubernetes.io/projected/b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62-kube-api-access-4hrn9\") pod \"maas-api-7bd8599656-m4c6c\" (UID: \"b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62\") " pod="odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c"
Sep 19 12:11:58.670591 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.670474    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f710bc83-dc05-4d3f-a314-f92271e2e8b9-maas-api-tls\") pod \"maas-api-6649958464-nszbc\" (UID: \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\") " pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc"
Sep 19 12:11:58.670591 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.670496    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62-openshift-service-ca\") pod \"maas-api-7bd8599656-m4c6c\" (UID: \"b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62\") " pod="odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c"
Sep 19 12:11:58.670591 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.670564    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f710bc83-dc05-4d3f-a314-f92271e2e8b9-metrics-certs\") pod \"maas-api-6649958464-nszbc\" (UID: \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\") " pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc"
Sep 19 12:11:58.670591 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.670586    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdrv2\" (UniqueName: \"kubernetes.io/projected/f710bc83-dc05-4d3f-a314-f92271e2e8b9-kube-api-access-bdrv2\") pod \"maas-api-6649958464-nszbc\" (UID: \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\") " pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc"
Sep 19 12:11:58.771280 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.771227    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hrn9\" (UniqueName: \"kubernetes.io/projected/b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62-kube-api-access-4hrn9\") pod \"maas-api-7bd8599656-m4c6c\" (UID: \"b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62\") " pod="odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c"
Sep 19 12:11:58.771483 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.771297    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f710bc83-dc05-4d3f-a314-f92271e2e8b9-maas-api-tls\") pod \"maas-api-6649958464-nszbc\" (UID: \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\") " pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc"
Sep 19 12:11:58.771483 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.771324    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62-openshift-service-ca\") pod \"maas-api-7bd8599656-m4c6c\" (UID: \"b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62\") " pod="odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c"
Sep 19 12:11:58.771483 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.771351    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f710bc83-dc05-4d3f-a314-f92271e2e8b9-metrics-certs\") pod \"maas-api-6649958464-nszbc\" (UID: \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\") " pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc"
Sep 19 12:11:58.771483 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.771369    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdrv2\" (UniqueName: \"kubernetes.io/projected/f710bc83-dc05-4d3f-a314-f92271e2e8b9-kube-api-access-bdrv2\") pod \"maas-api-6649958464-nszbc\" (UID: \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\") " pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc"
Sep 19 12:11:58.771663 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.771535    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f710bc83-dc05-4d3f-a314-f92271e2e8b9-openshift-service-ca\") pod \"maas-api-6649958464-nszbc\" (UID: \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\") " pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc"
Sep 19 12:11:58.771663 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.771587    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62-metrics-certs\") pod \"maas-api-7bd8599656-m4c6c\" (UID: \"b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62\") " pod="odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c"
Sep 19 12:11:58.771663 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.771651    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62-maas-api-tls\") pod \"maas-api-7bd8599656-m4c6c\" (UID: \"b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62\") " pod="odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c"
Sep 19 12:11:58.772259 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.772230    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62-openshift-service-ca\") pod \"maas-api-7bd8599656-m4c6c\" (UID: \"b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62\") " pod="odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c"
Sep 19 12:11:58.772396 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.772280    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f710bc83-dc05-4d3f-a314-f92271e2e8b9-openshift-service-ca\") pod \"maas-api-6649958464-nszbc\" (UID: \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\") " pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc"
Sep 19 12:11:58.774234 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.774211    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f710bc83-dc05-4d3f-a314-f92271e2e8b9-metrics-certs\") pod \"maas-api-6649958464-nszbc\" (UID: \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\") " pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc"
Sep 19 12:11:58.774353 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.774216    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62-metrics-certs\") pod \"maas-api-7bd8599656-m4c6c\" (UID: \"b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62\") " pod="odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c"
Sep 19 12:11:58.774690 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.774666    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f710bc83-dc05-4d3f-a314-f92271e2e8b9-maas-api-tls\") pod \"maas-api-6649958464-nszbc\" (UID: \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\") " pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc"
Sep 19 12:11:58.774752 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.774688    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62-maas-api-tls\") pod \"maas-api-7bd8599656-m4c6c\" (UID: \"b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62\") " pod="odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c"
Sep 19 12:11:58.781049 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.781023    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdrv2\" (UniqueName: \"kubernetes.io/projected/f710bc83-dc05-4d3f-a314-f92271e2e8b9-kube-api-access-bdrv2\") pod \"maas-api-6649958464-nszbc\" (UID: \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\") " pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc"
Sep 19 12:11:58.781168 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.781157    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hrn9\" (UniqueName: \"kubernetes.io/projected/b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62-kube-api-access-4hrn9\") pod \"maas-api-7bd8599656-m4c6c\" (UID: \"b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62\") " pod="odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c"
Sep 19 12:11:58.851544 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.851498    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc"
Sep 19 12:11:58.861728 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:58.861693    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c"
Sep 19 12:11:59.134346 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:59.134323    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6649958464-nszbc"]
Sep 19 12:11:59.136607 ip-10-0-133-209 kubenswrapper[2584]: W0919 12:11:59.136577    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf710bc83_dc05_4d3f_a314_f92271e2e8b9.slice/crio-9879ea48c14d9aeca584051a00117d86a3f00916bbbecad2666505c55a320a0f WatchSource:0}: Error finding container 9879ea48c14d9aeca584051a00117d86a3f00916bbbecad2666505c55a320a0f: Status 404 returned error can't find the container with id 9879ea48c14d9aeca584051a00117d86a3f00916bbbecad2666505c55a320a0f
Sep 19 12:11:59.138009 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:59.137990    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 12:11:59.148403 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:59.146246    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c"]
Sep 19 12:11:59.148523 ip-10-0-133-209 kubenswrapper[2584]: W0919 12:11:59.148497    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69d00ec_cb7c_44ef_a677_ab6e3ee8eb62.slice/crio-7696bd0c44ed2b5551d037ceb9c93dd0df8be480de0a3b5b58646a420bb1ab82 WatchSource:0}: Error finding container 7696bd0c44ed2b5551d037ceb9c93dd0df8be480de0a3b5b58646a420bb1ab82: Status 404 returned error can't find the container with id 7696bd0c44ed2b5551d037ceb9c93dd0df8be480de0a3b5b58646a420bb1ab82
Sep 19 12:11:59.531307 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:59.531204    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c" event={"ID":"b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62","Type":"ContainerStarted","Data":"7696bd0c44ed2b5551d037ceb9c93dd0df8be480de0a3b5b58646a420bb1ab82"}
Sep 19 12:11:59.532462 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:11:59.532431    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc" event={"ID":"f710bc83-dc05-4d3f-a314-f92271e2e8b9","Type":"ContainerStarted","Data":"9879ea48c14d9aeca584051a00117d86a3f00916bbbecad2666505c55a320a0f"}
Sep 19 12:12:00.540332 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:00.540288    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc" event={"ID":"f710bc83-dc05-4d3f-a314-f92271e2e8b9","Type":"ContainerStarted","Data":"8d909b937e0ac867b79c9a5bae91cfd4d76953dd5e1fff403d4c5e8ba0634537"}
Sep 19 12:12:00.540986 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:00.540362    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc"
Sep 19 12:12:00.541948 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:00.541894    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c" event={"ID":"b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62","Type":"ContainerStarted","Data":"4e51ce4fe5b3a412390ec5138ed8606bb3f0ede1b8b2ca0d9be35236f569d403"}
Sep 19 12:12:00.542077 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:00.541979    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c"
Sep 19 12:12:00.562877 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:00.562814    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc" podStartSLOduration=2.099982049 podStartE2EDuration="2.562796838s" totalImagesPullingTime="462.814789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:12:00.560296249 +0000 UTC m=+2304.760326053" watchObservedRunningTime="2026-09-19 12:12:00.562796838 +0000 UTC m=+2304.762826675"
Sep 19 12:12:00.577436 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:00.577372    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c" podStartSLOduration=2.174053387 podStartE2EDuration="2.577354106s" totalImagesPullingTime="403.300719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 12:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:12:00.576221139 +0000 UTC m=+2304.776250947" watchObservedRunningTime="2026-09-19 12:12:00.577354106 +0000 UTC m=+2304.777383926"
Sep 19 12:12:06.552820 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:06.552785    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc"
Sep 19 12:12:06.553268 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:06.553146    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7bd8599656-m4c6c"
Sep 19 12:12:06.615032 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:06.614985    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6649958464-nszbc"]
Sep 19 12:12:06.615284 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:06.615236    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc" podUID="f710bc83-dc05-4d3f-a314-f92271e2e8b9" containerName="maas-api" containerID="cri-o://8d909b937e0ac867b79c9a5bae91cfd4d76953dd5e1fff403d4c5e8ba0634537" gracePeriod=30
Sep 19 12:12:06.865451 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:06.865413    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc"
Sep 19 12:12:06.949377 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:06.949341    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f710bc83-dc05-4d3f-a314-f92271e2e8b9-maas-api-tls\") pod \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\" (UID: \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\") "
Sep 19 12:12:06.949584 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:06.949411    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f710bc83-dc05-4d3f-a314-f92271e2e8b9-openshift-service-ca\") pod \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\" (UID: \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\") "
Sep 19 12:12:06.949584 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:06.949434    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f710bc83-dc05-4d3f-a314-f92271e2e8b9-metrics-certs\") pod \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\" (UID: \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\") "
Sep 19 12:12:06.949584 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:06.949546    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdrv2\" (UniqueName: \"kubernetes.io/projected/f710bc83-dc05-4d3f-a314-f92271e2e8b9-kube-api-access-bdrv2\") pod \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\" (UID: \"f710bc83-dc05-4d3f-a314-f92271e2e8b9\") "
Sep 19 12:12:06.949838 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:06.949814    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f710bc83-dc05-4d3f-a314-f92271e2e8b9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f710bc83-dc05-4d3f-a314-f92271e2e8b9" (UID: "f710bc83-dc05-4d3f-a314-f92271e2e8b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 12:12:06.949945 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:06.949895    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f710bc83-dc05-4d3f-a314-f92271e2e8b9-openshift-service-ca\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:12:06.951977 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:06.951952    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f710bc83-dc05-4d3f-a314-f92271e2e8b9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f710bc83-dc05-4d3f-a314-f92271e2e8b9" (UID: "f710bc83-dc05-4d3f-a314-f92271e2e8b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 12:12:06.952088 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:06.951985    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f710bc83-dc05-4d3f-a314-f92271e2e8b9-kube-api-access-bdrv2" (OuterVolumeSpecName: "kube-api-access-bdrv2") pod "f710bc83-dc05-4d3f-a314-f92271e2e8b9" (UID: "f710bc83-dc05-4d3f-a314-f92271e2e8b9"). InnerVolumeSpecName "kube-api-access-bdrv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 12:12:06.952088 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:06.952042    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f710bc83-dc05-4d3f-a314-f92271e2e8b9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f710bc83-dc05-4d3f-a314-f92271e2e8b9" (UID: "f710bc83-dc05-4d3f-a314-f92271e2e8b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 12:12:07.050817 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:07.050766    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f710bc83-dc05-4d3f-a314-f92271e2e8b9-maas-api-tls\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:12:07.050817 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:07.050811    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f710bc83-dc05-4d3f-a314-f92271e2e8b9-metrics-certs\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:12:07.050817 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:07.050826    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdrv2\" (UniqueName: \"kubernetes.io/projected/f710bc83-dc05-4d3f-a314-f92271e2e8b9-kube-api-access-bdrv2\") on node \"ip-10-0-133-209.ec2.internal\" DevicePath \"\""
Sep 19 12:12:07.579046 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:07.579007    2584 generic.go:356] "Generic (PLEG): container finished" podID="f710bc83-dc05-4d3f-a314-f92271e2e8b9" containerID="8d909b937e0ac867b79c9a5bae91cfd4d76953dd5e1fff403d4c5e8ba0634537" exitCode=0
Sep 19 12:12:07.579532 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:07.579071    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc"
Sep 19 12:12:07.579532 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:07.579083    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc" event={"ID":"f710bc83-dc05-4d3f-a314-f92271e2e8b9","Type":"ContainerDied","Data":"8d909b937e0ac867b79c9a5bae91cfd4d76953dd5e1fff403d4c5e8ba0634537"}
Sep 19 12:12:07.579532 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:07.579166    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6649958464-nszbc" event={"ID":"f710bc83-dc05-4d3f-a314-f92271e2e8b9","Type":"ContainerDied","Data":"9879ea48c14d9aeca584051a00117d86a3f00916bbbecad2666505c55a320a0f"}
Sep 19 12:12:07.579532 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:07.579189    2584 scope.go:117] "RemoveContainer" containerID="8d909b937e0ac867b79c9a5bae91cfd4d76953dd5e1fff403d4c5e8ba0634537"
Sep 19 12:12:07.589187 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:07.589163    2584 scope.go:117] "RemoveContainer" containerID="8d909b937e0ac867b79c9a5bae91cfd4d76953dd5e1fff403d4c5e8ba0634537"
Sep 19 12:12:07.589537 ip-10-0-133-209 kubenswrapper[2584]: E0919 12:12:07.589516    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d909b937e0ac867b79c9a5bae91cfd4d76953dd5e1fff403d4c5e8ba0634537\": container with ID starting with 8d909b937e0ac867b79c9a5bae91cfd4d76953dd5e1fff403d4c5e8ba0634537 not found: ID does not exist" containerID="8d909b937e0ac867b79c9a5bae91cfd4d76953dd5e1fff403d4c5e8ba0634537"
Sep 19 12:12:07.589628 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:07.589551    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d909b937e0ac867b79c9a5bae91cfd4d76953dd5e1fff403d4c5e8ba0634537"} err="failed to get container status \"8d909b937e0ac867b79c9a5bae91cfd4d76953dd5e1fff403d4c5e8ba0634537\": rpc error: code = NotFound desc = could not find container \"8d909b937e0ac867b79c9a5bae91cfd4d76953dd5e1fff403d4c5e8ba0634537\": container with ID starting with 8d909b937e0ac867b79c9a5bae91cfd4d76953dd5e1fff403d4c5e8ba0634537 not found: ID does not exist"
Sep 19 12:12:07.600621 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:07.600588    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6649958464-nszbc"]
Sep 19 12:12:07.604091 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:07.604063    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6649958464-nszbc"]
Sep 19 12:12:07.904674 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:07.904591    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f710bc83-dc05-4d3f-a314-f92271e2e8b9" path="/var/lib/kubelet/pods/f710bc83-dc05-4d3f-a314-f92271e2e8b9/volumes"
Sep 19 12:12:48.073231 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:48.073187    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-7645b9c6f5-cn92z_820ea15c-9147-496e-86bd-479a0bdb5def/manager/0.log"
Sep 19 12:12:48.560194 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:48.560164    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-77df98bf98-n72lf_0dce098a-bdf6-4503-a8c2-d0d82b0911dd/server/0.log"
Sep 19 12:12:49.127086 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:49.127055    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7bd8599656-m4c6c_b69d00ec-cb7c-44ef-a677-ab6e3ee8eb62/maas-api/0.log"
Sep 19 12:12:50.439534 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:50.439507    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b449b7679-7bz44_7828b099-93eb-4f9f-84c0-59b1f6c52000/manager/0.log"
Sep 19 12:12:50.672334 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:50.672301    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-54b6d5c989-9kz6w_b90272ae-624f-4c2f-a9f0-aee7c565f4f3/manager/0.log"
Sep 19 12:12:50.906686 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:50.906655    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6d9b9f85cb-ks2j6_6eb955af-5023-4977-b253-ce98d2950e19/manager/0.log"
Sep 19 12:12:51.609463 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:51.609427    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864788f75f-x6j8q_f5e9cbc1-01a5-4d28-8287-f100f26a76af/kube-auth-proxy/0.log"
Sep 19 12:12:52.205666 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:52.205632    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-687866bb-9xvr5_5b388e80-43fb-4367-a7c1-6c53f04caffa/router/0.log"
Sep 19 12:12:52.791125 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:52.791094    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-698db7d589-zjppm_d01fa8b2-646f-4958-8b31-46c250d755f5/main/0.log"
Sep 19 12:12:52.905003 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:52.904967    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-5bccb7bbf7-9lgv8_aff31354-fd81-4741-860a-2506c059d365/main/0.log"
Sep 19 12:12:58.482766 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:58.482733    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:12:58.553107 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:58.553049    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:12:59.179013 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:12:59.178967    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:13:01.335305 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:01.335268    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dpgkd_f96896a2-5494-4bc6-81c9-154860ee5f6b/global-pull-secret-syncer/0.log"
Sep 19 12:13:01.423408 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:01.423373    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log"
Sep 19 12:13:01.542669 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:01.542631    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-209.ec2.internal_49bc127c496c8981768be48e6fe5a22e/haproxy/0.log"
Sep 19 12:13:06.858908 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:06.858868    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b449b7679-7bz44_7828b099-93eb-4f9f-84c0-59b1f6c52000/manager/0.log"
Sep 19 12:13:06.955009 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:06.954974    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-54b6d5c989-9kz6w_b90272ae-624f-4c2f-a9f0-aee7c565f4f3/manager/0.log"
Sep 19 12:13:07.058272 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:07.058238    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6d9b9f85cb-ks2j6_6eb955af-5023-4977-b253-ce98d2950e19/manager/0.log"
Sep 19 12:13:08.497260 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:08.497200    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-nll25_53ad775e-63df-4c3d-856d-f17723aed41e/cluster-monitoring-operator/0.log"
Sep 19 12:13:08.843226 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:08.843189    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fv299_4c7a5a73-a310-4974-9fb9-58220b1ae2c8/node-exporter/0.log"
Sep 19 12:13:08.861226 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:08.861194    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fv299_4c7a5a73-a310-4974-9fb9-58220b1ae2c8/kube-rbac-proxy/0.log"
Sep 19 12:13:08.880542 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:08.880504    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fv299_4c7a5a73-a310-4974-9fb9-58220b1ae2c8/init-textfile/0.log"
Sep 19 12:13:09.900695 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:09.900648    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dktw9_ee2c4ee6-287a-4768-9cba-cb8729cef657/konnectivity-agent/0.log" limit=1024
Sep 19 12:13:10.784818 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:10.784790    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-r6svd_83c3531f-3ddd-45b9-91cd-3ac141ebd487/networking-console-plugin/0.log"
Sep 19 12:13:11.250430 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.250391    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"]
Sep 19 12:13:11.250881 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.250814    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f710bc83-dc05-4d3f-a314-f92271e2e8b9" containerName="maas-api"
Sep 19 12:13:11.250881 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.250828    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f710bc83-dc05-4d3f-a314-f92271e2e8b9" containerName="maas-api"
Sep 19 12:13:11.250988 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.250959    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f710bc83-dc05-4d3f-a314-f92271e2e8b9" containerName="maas-api"
Sep 19 12:13:11.253900 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.253516    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"
Sep 19 12:13:11.256486 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.256466    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fgjdw\"/\"openshift-service-ca.crt\""
Sep 19 12:13:11.256585 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.256478    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fgjdw\"/\"kube-root-ca.crt\""
Sep 19 12:13:11.257806 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.257785    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fgjdw\"/\"default-dockercfg-6b6xz\""
Sep 19 12:13:11.262661 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.262636    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"]
Sep 19 12:13:11.332316 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.332274    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/75fde399-f5e0-4b48-8a32-c879c565abd1-podres\") pod \"perf-node-gather-daemonset-rptj9\" (UID: \"75fde399-f5e0-4b48-8a32-c879c565abd1\") " pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"
Sep 19 12:13:11.332534 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.332338    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjhbp\" (UniqueName: \"kubernetes.io/projected/75fde399-f5e0-4b48-8a32-c879c565abd1-kube-api-access-sjhbp\") pod \"perf-node-gather-daemonset-rptj9\" (UID: \"75fde399-f5e0-4b48-8a32-c879c565abd1\") " pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"
Sep 19 12:13:11.332534 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.332419    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75fde399-f5e0-4b48-8a32-c879c565abd1-sys\") pod \"perf-node-gather-daemonset-rptj9\" (UID: \"75fde399-f5e0-4b48-8a32-c879c565abd1\") " pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"
Sep 19 12:13:11.332534 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.332481    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75fde399-f5e0-4b48-8a32-c879c565abd1-lib-modules\") pod \"perf-node-gather-daemonset-rptj9\" (UID: \"75fde399-f5e0-4b48-8a32-c879c565abd1\") " pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"
Sep 19 12:13:11.332654 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.332617    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/75fde399-f5e0-4b48-8a32-c879c565abd1-proc\") pod \"perf-node-gather-daemonset-rptj9\" (UID: \"75fde399-f5e0-4b48-8a32-c879c565abd1\") " pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"
Sep 19 12:13:11.433044 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.432999    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75fde399-f5e0-4b48-8a32-c879c565abd1-sys\") pod \"perf-node-gather-daemonset-rptj9\" (UID: \"75fde399-f5e0-4b48-8a32-c879c565abd1\") " pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"
Sep 19 12:13:11.433240 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.433061    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75fde399-f5e0-4b48-8a32-c879c565abd1-lib-modules\") pod \"perf-node-gather-daemonset-rptj9\" (UID: \"75fde399-f5e0-4b48-8a32-c879c565abd1\") " pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"
Sep 19 12:13:11.433240 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.433114    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/75fde399-f5e0-4b48-8a32-c879c565abd1-proc\") pod \"perf-node-gather-daemonset-rptj9\" (UID: \"75fde399-f5e0-4b48-8a32-c879c565abd1\") " pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"
Sep 19 12:13:11.433240 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.433131    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/75fde399-f5e0-4b48-8a32-c879c565abd1-podres\") pod \"perf-node-gather-daemonset-rptj9\" (UID: \"75fde399-f5e0-4b48-8a32-c879c565abd1\") " pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"
Sep 19 12:13:11.433240 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.433148    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjhbp\" (UniqueName: \"kubernetes.io/projected/75fde399-f5e0-4b48-8a32-c879c565abd1-kube-api-access-sjhbp\") pod \"perf-node-gather-daemonset-rptj9\" (UID: \"75fde399-f5e0-4b48-8a32-c879c565abd1\") " pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"
Sep 19 12:13:11.433240 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.433143    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75fde399-f5e0-4b48-8a32-c879c565abd1-sys\") pod \"perf-node-gather-daemonset-rptj9\" (UID: \"75fde399-f5e0-4b48-8a32-c879c565abd1\") " pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"
Sep 19 12:13:11.433240 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.433218    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/75fde399-f5e0-4b48-8a32-c879c565abd1-proc\") pod \"perf-node-gather-daemonset-rptj9\" (UID: \"75fde399-f5e0-4b48-8a32-c879c565abd1\") " pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"
Sep 19 12:13:11.433450 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.433242    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75fde399-f5e0-4b48-8a32-c879c565abd1-lib-modules\") pod \"perf-node-gather-daemonset-rptj9\" (UID: \"75fde399-f5e0-4b48-8a32-c879c565abd1\") " pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"
Sep 19 12:13:11.433450 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.433285    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/75fde399-f5e0-4b48-8a32-c879c565abd1-podres\") pod \"perf-node-gather-daemonset-rptj9\" (UID: \"75fde399-f5e0-4b48-8a32-c879c565abd1\") " pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"
Sep 19 12:13:11.441124 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.441092    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjhbp\" (UniqueName: \"kubernetes.io/projected/75fde399-f5e0-4b48-8a32-c879c565abd1-kube-api-access-sjhbp\") pod \"perf-node-gather-daemonset-rptj9\" (UID: \"75fde399-f5e0-4b48-8a32-c879c565abd1\") " pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"
Sep 19 12:13:11.455297 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.455264    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/1.log"
Sep 19 12:13:11.466743 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.466712    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4bxfj_24aae5c7-e112-496e-ba52-a52e02e520c1/console-operator/2.log"
Sep 19 12:13:11.564945 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.564872    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"
Sep 19 12:13:11.839762 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.839737    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"]
Sep 19 12:13:11.842743 ip-10-0-133-209 kubenswrapper[2584]: W0919 12:13:11.842715    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod75fde399_f5e0_4b48_8a32_c879c565abd1.slice/crio-abac681b2d874e473139e92b9cf271f24a086bb84854c72f902b00eedb6b24d1 WatchSource:0}: Error finding container abac681b2d874e473139e92b9cf271f24a086bb84854c72f902b00eedb6b24d1: Status 404 returned error can't find the container with id abac681b2d874e473139e92b9cf271f24a086bb84854c72f902b00eedb6b24d1
Sep 19 12:13:11.904031 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:11.904000    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9" event={"ID":"75fde399-f5e0-4b48-8a32-c879c565abd1","Type":"ContainerStarted","Data":"abac681b2d874e473139e92b9cf271f24a086bb84854c72f902b00eedb6b24d1"}
Sep 19 12:13:12.635833 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:12.635805    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-9zcpw_40987fee-2f7a-4681-a32c-22aaa7b8a8f7/volume-data-source-validator/0.log"
Sep 19 12:13:12.910697 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:12.910591    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9" event={"ID":"75fde399-f5e0-4b48-8a32-c879c565abd1","Type":"ContainerStarted","Data":"59ff432f6b3945abd980b9583f70970f47a8022540cc7555699e53021cb73393"}
Sep 19 12:13:12.910697 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:12.910672    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"
Sep 19 12:13:12.925701 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:12.925640    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9" podStartSLOduration=1.925618439 podStartE2EDuration="1.925618439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 12:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 12:13:12.925347536 +0000 UTC m=+2377.125377339" watchObservedRunningTime="2026-09-19 12:13:12.925618439 +0000 UTC m=+2377.125648222"
Sep 19 12:13:13.705941 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:13.705892    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t27gs_7b681f4b-5ee2-4693-8e41-4be4f0a374c1/dns/0.log"
Sep 19 12:13:13.726338 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:13.726302    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t27gs_7b681f4b-5ee2-4693-8e41-4be4f0a374c1/kube-rbac-proxy/0.log"
Sep 19 12:13:13.791084 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:13.791054    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vc5vt_869b7a50-52b6-4167-aa2d-f5152f2c8eb4/dns-node-resolver/0.log"
Sep 19 12:13:14.339268 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:14.339230    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rsnfw_b2e7da35-ed65-4cd7-8d44-9b25ff53fba2/node-ca/0.log"
Sep 19 12:13:16.161721 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:16.161687    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864788f75f-x6j8q_f5e9cbc1-01a5-4d28-8287-f100f26a76af/kube-auth-proxy/0.log"
Sep 19 12:13:16.316391 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:16.316357    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-687866bb-9xvr5_5b388e80-43fb-4367-a7c1-6c53f04caffa/router/0.log"
Sep 19 12:13:17.219292 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:17.219258    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h4xjv_2066d78e-223c-4531-98aa-edc826b41c3b/serve-healthcheck-canary/0.log"
Sep 19 12:13:17.714979 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:17.714940    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-7wgwd_cc7e845c-b6fd-4a96-8258-80b4e8f02e57/insights-operator/0.log"
Sep 19 12:13:17.715844 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:17.715814    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-7wgwd_cc7e845c-b6fd-4a96-8258-80b4e8f02e57/insights-operator/1.log"
Sep 19 12:13:17.851031 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:17.850997    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t2fzm_047fafca-cf10-4883-aad0-0c68166d05e0/kube-rbac-proxy/0.log"
Sep 19 12:13:17.869295 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:17.869263    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t2fzm_047fafca-cf10-4883-aad0-0c68166d05e0/exporter/0.log"
Sep 19 12:13:17.888736 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:17.888706    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t2fzm_047fafca-cf10-4883-aad0-0c68166d05e0/extractor/0.log"
Sep 19 12:13:18.927130 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:18.927106    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fgjdw/perf-node-gather-daemonset-rptj9"
Sep 19 12:13:20.292443 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:20.292408    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-9f59f68b4-vx8jw_549eeb37-540c-4280-9b6d-87a08afaddfa/manager/0.log"
Sep 19 12:13:20.311023 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:20.310995    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-9f59f68b4-vx8jw_549eeb37-540c-4280-9b6d-87a08afaddfa/copy-manifests/0.log"
Sep 19 12:13:21.457497 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:21.457471    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864788f75f-x6j8q_f5e9cbc1-01a5-4d28-8287-f100f26a76af/kube-auth-proxy/0.log"
Sep 19 12:13:21.575150 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:21.575073    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-687866bb-9xvr5_5b388e80-43fb-4367-a7c1-6c53f04caffa/router/0.log"
Sep 19 12:13:22.183874 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:22.183845    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-7645b9c6f5-cn92z_820ea15c-9147-496e-86bd-479a0bdb5def/manager/0.log"
Sep 19 12:13:22.659403 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:22.659313    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-77df98bf98-n72lf_0dce098a-bdf6-4503-a8c2-d0d82b0911dd/server/0.log"
Sep 19 12:13:24.101496 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:24.101466    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864788f75f-x6j8q_f5e9cbc1-01a5-4d28-8287-f100f26a76af/kube-auth-proxy/0.log"
Sep 19 12:13:24.230980 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:24.230946    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-687866bb-9xvr5_5b388e80-43fb-4367-a7c1-6c53f04caffa/router/0.log"
Sep 19 12:13:27.684976 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:27.684941    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-ftf42_aa857849-d4f4-4815-aa2d-8412e4a8033a/kube-storage-version-migrator-operator/1.log"
Sep 19 12:13:27.686819 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:27.686794    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-ftf42_aa857849-d4f4-4815-aa2d-8412e4a8033a/kube-storage-version-migrator-operator/0.log"
Sep 19 12:13:28.759853 ip-10-0-133-209 kubenswrapper[2584]: I0919 12:13:28.759821    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6pbcp_11fdca9a-6699-4e5e-b55e-78ca9308ed2e/kube-multus/0.log"